phase: 162_search_recall_small_sources
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)** - Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched. - Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated. - Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched). - Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`). - Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382. - `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**. - `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**. - Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness. - Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB. - Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.) - Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass. - Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`". - Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
This commit is contained in:
@@ -0,0 +1,70 @@
|
||||
# Phase 162 — Search recall: accent-folded name hits & grep, KB-overview source inventory, search-discipline teaching
|
||||
|
||||
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, shared conversation "How do I build a Besta?" (share token `58525626-4462-4baf-bcf2-3abc81f18284`). The owner asked "Why did you miss it? Let's try to improve your search techniques and system prompt." The agent's post-mortem identified **four stacked failures**, three of which are code-fixable:
|
||||
|
||||
1. **The `<knowledge_base>` outline is lossy and is the only map the model gets.** The outline listed 3 groups (Gufo, Brain-of-Reese, Deployments); the live index had 14 sources. `ikea_instructions` (1 doc) and `players_handbook` (1 doc) were **invisible** — a whole class of source-level questions could not be surfaced by the outline at all. The model answered from the outline instead of running `ls`.
|
||||
2. **Notes poisoned the model with a true-but-different fact** (the phase-140/148 note "the IKEA tar is gitignored, on disk only" was over-generalized to "the brain has no copy of this PDF"). Model behavior — mitigated here by prompt teaching (D4), plus an owner runtime follow-up (A6, recorded, not code).
|
||||
3. **The Å/ASCII accent-folding gap.** `grep "besta"` returned only phase docs and never the PDF, because the only product name in the PDF's text layer is `BESTÅ` (U+00C5): the filename is ASCII (`besta-frame-white__…`) but the content is not, and the exact-substring grep cannot bridge that. On the name-hit side the gap is symmetric: the question token `BESTÅ` normalizes (lower + alnum-only) to `best` — not `besta` — so an accented question never name-matches the ASCII filename's stem sub-components.
|
||||
4. **Retrieval under-ranked the PDF (the phase-119 failure class).** The doc's title is the generic "Assembly Instructions" (title ranking gives nothing); by phase-119 LOCKED A2 the digitless token `besta` (≥4 chars) equals a stem sub-component of `besta-frame-white__AA-1272068-11-100` and should have name-matched, but the name-hit row lost the fused walk to the phase docs whose bodies literally contain "besta". Flagged by the post-mortem as a hypothesis needing a test — **this phase pins it with the battery (D5)** rather than assuming a finding.
|
||||
|
||||
**Story:** n/a (owner-directed RAG behavior change; extends the phase-119 name-hit mechanism, the phase-31 KB overview, the phase-68/70 grep, and the phase-117 prompt pins).
|
||||
|
||||
## Objective
|
||||
Accented names (`BESTÅ` ↔ `besta`) are found **both ways** — by the name-hit path (question side) and by `grep` (content side); **no source can be invisible** to `<knowledge_base>` (a deterministic, non-LLM source-inventory line is appended to the stored overview); the model is **taught the search discipline** the post-mortem adopted (outline ≠ inventory; `ls` first; part-number/filename-token greps; gitignored ≠ unindexed); and the battery question "How do I build a Besta?" **pins** the IKEA doc into the suggested top-5 — turning the anecdote into a regression test.
|
||||
|
||||
## Dependencies
|
||||
- `119_name_signal_read_chips` (complete) — the phase-119 name-hit rule (LOCKED A2), the `name_hit` flag, the bonus (LOCKED A3), and the `suggested:` eval-battery line this phase extends.
|
||||
- `31`-line KB overview (complete; `app/rag/overview.py`) — the `regenerate_overview` best-effort/change-gated generation this phase appends the inventory to.
|
||||
- `68_search_tool` / `70_harness_aligned_tools` (complete) — the `grep` contract (case-insensitive fixed substring, locator, teaching refusals) this phase extends.
|
||||
- `117_tool_call_compact` (complete) — the `TOOLS_SECTION` byte-pins this phase re-cuts.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
**D1 — accent-folded name normalization (task 01, LOCKED A1).** One new helper in `app/rag/retriever.py`: `_fold(s)` = `unicodedata` NFKD normalize + drop every char with a non-zero combining class (stdlib only; no new dependency). `_normalize_name` becomes the existing `[a-z0-9]+` scan of `_fold(s.lower())`. Because `_normalize_name` is the **single** normalization both sides of name matching pass through — the question tokens in `name_hit_tokens` AND the path parts / file stem / stem sub-components in `_name_parts` — the one change bridges both directions: the ASCII question `besta` keeps matching the ASCII filename's sub-components (unchanged behavior), the accented question `BESTÅ` now normalizes to `besta` (was `best` — the 2026-10-07 gap), and an accented **filename** (`BÅsta.md` → `basta`) matches an ASCII question. Folding cases the tests pin: `Å/å → a`, `Ä/ä/Ö/ö/Ü/ü → a/o/u` (the letter, decomposition preserved), `ß → ss`, the `fi` ligature → `fi`, `ç → c`; characters with no NFKD decomposition (CJK, `①`) still drop out under the alnum scan, exactly as today. **Everything else in the A2 mechanism is untouched**: the digit-bearing-prefix / digitless-equality match classes, titles-never-matched, the (matched-token count, catalog order) ranking, `NAME_TOKEN_MIN_LEN`, the versioned-name join, `NAME_HIT_LIMIT`, the `name_hit` flag, the phase-119 bonus (LOCKED A3), and the FTS / cosine / A8 gate (LOCKED A7/A8 — the `tsvector` 'english' pipeline is NOT modified; same-accents FTS already bridges, accent-vs-ASCII FTS bridging is out of scope by the lock).
|
||||
|
||||
**D2 — the KB-overview source inventory (task 02, LOCKED A2).** The overview is `lite`-generated and lossy (D1-failure #1); the fix is **deterministic, non-LLM**: `regenerate_overview` (happy path only) appends one line to the stored content, after the model's outline: `Sources: <name> (<k> doc|docs), …` — one entry per DISTINCT source, catalog order (`source, path`), exact per-source document counts. Computed from the same catalogue rows the prompt builder already reads (no new query). `build_overview_prompt` / `OVERVIEW_INSTRUCTION` are **byte-identical** (the model never sees the tail — it is mechanical, not a model burden); `load_kb_overview` / `build_kb_section` are untouched (the existing longest-prefix cap applies to the whole stored text). Budget: the model caps the outline at ~1500 chars per the instruction; one inventory line is ~15 chars/source — at the default `BOR_KB_OVERVIEW_MAX_CHARS` (4000) the tail always fits; a smaller owner-set cap cuts the tail first (accepted; the cap is owner-tunable, the record here is the constraint). The zero-doc early return and the `LLMError` path are **byte-identical** (the previous row — tail included, it is part of the stored content — stays intact; an old outline is better than none).
|
||||
|
||||
**D3 — accent-folding grep (task 03, LOCKED A3).** `grep_document` in `app/rag/agent.py` folds **both sides** — the pattern and each content line — through the SAME `_fold` as D1 (imported from `app.rag.retriever`, which `agent.py` already imports from: grep and the name-hit path can never drift apart) plus lowercase (replacing the current lower-only comparison): `grep "besta"` now matches a content line `The BESTÅ frame`. The contract remains a **fixed substring** (the phase-68 owner lock) — the fold changes the alphabet it ignores, not the matching semantics; still a locator, still ≤20 lines, still line-truncated at 200 chars. The regex teaching (`NO_MATCHES_REGEX` / `_SCOPED`, `looks_like_regex`, the plain-form retry hint) is **byte-identical** — the plain-substring claim it teaches is still true. Copy updates (all in this task): the `agent.py` module docstring grep paragraph, the `AGENT_TOOLS` grep function description + the `pattern` parameter description, and the `TOOLS_SECTION` grep sentence in `app/rag/prompts.py` ("case-insensitive" → "case- and accent-insensitive"), plus D4's new sentence. The phase-117 `test_prompt_lock.py` / `test_prompts.py` / `test_agent.py` pins re-cut in the same task (the phase-124 pattern: copy change + re-cut pins in one task).
|
||||
|
||||
**D4 — the search-discipline sentence (task 03, LOCKED A4).** Appended to the `TOOLS_SECTION` tools paragraph **immediately after the grep sentence** (the conversation's placement), LOCKED verbatim:
|
||||
|
||||
> `A summary or outline is a hint, never an inventory: before claiming the knowledge base lacks something, run `ls` with no `path` to list every source, and `grep` the artifact's part number or a filename token. `grep` folds accents — `besta` matches `BESTÅ` — but one pattern is one string: a name's split tokens (`frame`, `white`) and its part number (`AA-1272068`) need separate greps. A note saying a file is gitignored describes its repo status, not whether it is indexed as a source.`
|
||||
|
||||
(Adapted from the conversation's proposed patch for the D3 fold — the conversation's original sentence assumed a non-folding grep; the owner asked for "improve your search techniques and system prompt", which is the permission source for this copy change.) `PERSONA`, the LOW/`DEFLECT_MODE` body, and every other `TOOLS_SECTION` sentence stay byte-identical. The mock LLM is unaffected (it keys on markers, not on the sentence).
|
||||
|
||||
**D5 — verification (task 04).** A new dedicated Playwright suite `tests/e2e/test_search_recall_small_sources.py` (run in isolation; the phase-119 local-directory-source fixture pattern) with a synthetic replica of the live miss: source `ikea_instructions` (ONE doc, `besta-frame-white__AA-1272068-11-100.md`, body containing the name **only** in accented form — no ASCII "besta") + 2–3 distractor docs in other sources whose bodies contain ASCII "besta" (the phase-docs stand-in). The **pre-fix evidence is measured and recorded** in the task file: with the task-01 fold temporarily reverted (git stash), the ACCENTED question "How do I build the BESTÅ?" must lose the accented doc from the suggested top-5 (record the measured ranks); restore the fix. Gated assertions: (1) the accented question's `done`-frame `suggestions` include the `ikea_instructions` doc (the D1 question-side fold); (2) the ASCII question "How do I build a Besta?" includes it too (the phase-119 path regression); (3) the stored `kb_overview` row (read through the fixture session) ends with the `Sources: …` tail naming **every** fixture source with exact counts — the 1-doc source visible (D2); (4) the out-of-KB question's gate verdict is unchanged (D1–D3 must not move the A8 gate). `tests/fixtures/retrieval_battery.txt` (the tracked phase-119 battery) gains `How do I build a Besta?` with its gate comment: on the 1,000-doc live replica (the `restore-test-db` skill; the dump is a local gitignored artifact — if missing, STOP and ask the owner, never fabricate numbers) the battery runs when the replica's `documents` table contains an `ikea_instructions`-class source (check `SELECT DISTINCT source FROM documents`) and the suggested top-5 must include `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`; if the replica lacks it (the snapshot may predate the import), the five existing phase-119 questions still run (their gate must NOT move — the name-hit change is behavior-preserving for them) and the Besta question is recorded as **skipped loudly (source absent in the replica snapshot)** — the phase-140 "skipped loudly" precedent. The under-rank hypothesis (D1-failure #4) is thus either pinned green or handed to the owner with measured ranks — never assumed.
|
||||
|
||||
**Invariants (all tasks).** The A8 honesty gate, the FTS `tsvector` pipeline, the cosine path, every SSE frame shape, the LOW prompt, `PERSONA`, the `NO_MATCHES_REGEX`/`_SCOPED` teaching lines, the mock-LLM markers, and the `build_overview_prompt` / `OVERVIEW_INSTRUCTION` inputs are byte-identical. `PLAN.md` untouched (Protocol B — A5).
|
||||
|
||||
## Tasks
|
||||
1. `01_accent_fold_name_norms.md` — the shared `_fold` helper + the `_normalize_name` fold (D1).
|
||||
2. `02_kb_overview_source_inventory.md` — the deterministic `Sources: …` tail (D2).
|
||||
3. `03_grep_folding_search_discipline.md` — the folded `grep_document` + the three copy updates + the D4 sentence + the re-cut pins (D3+D4).
|
||||
4. `04_e2e_battery_pins.md` — the dedicated E2E, the pre-fix evidence, the battery question + replica run, the final gates (D5).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_retriever.py` (D1: the bridge both ways, the fold cases, the A2 non-regression set), `tests/unit/test_overview.py` (D2: the tail format/order/counts, the 1-doc survival, the zero-doc + LLM-failure paths, the byte-identical `build_overview_prompt`), `tests/unit/test_agent.py` (D3: `grep_document` fold both sides, the scoped + unscoped paths, the copy pins), `tests/unit/test_prompts.py` + `tests/unit/test_prompt_lock.py` (D3+D4: the re-cut pins; the LOW + `PERSONA` byte-identity).
|
||||
- E2E: `tests/e2e/test_search_recall_small_sources.py`, **in isolation** (`uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov`); the phase-119/117/31 regression suites green (the re-cut pins included).
|
||||
- Battery (task 04): per D5 — the replica run recorded in the task file; the five phase-119 questions' gate unchanged.
|
||||
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `grep "besta"` (unit + the E2E fixture's content layer) matches an accented `BESTÅ` content line, and the accented question "How do I build the BESTÅ?" name-matches the ASCII filename's stem sub-components — the fold is one function, both sides, measured (the pre-fix ranks recorded in `04_…md`).
|
||||
- [ ] The stored `kb_overview` row ends with the deterministic `Sources: …` line naming every source with exact counts; a 1-doc source is visible; the zero-doc and LLM-failure paths are byte-identical to the pre-phase behavior.
|
||||
- [ ] The `TOOLS_SECTION` carries the folded grep contract + the LOCKED D4 sentence (all other pins byte-identical; the re-cut pins green).
|
||||
- [ ] `tests/e2e/test_search_recall_small_sources.py` green in isolation; the battery question is recorded per D5; `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass (the grep contract line in `AGENTS.md`'s §6 quick reference and `PLAN.md` §6's tool table mention "case-insensitive fixed substring" — the final pass updates `AGENTS.md`'s tool description to "case- and accent-insensitive"; `PLAN.md` stays untouched per A5, folded at the next redo).
|
||||
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — the fold**: NFKD normalize + combining-class strip (stdlib `unicodedata`) applied inside `_normalize_name` — the single normalization both sides of name matching use; the A2 match classes, ranking, caps, flags, the phase-119 bonus, and the FTS/cosine/A8 pipeline are untouched (owner-confirmed 2026-10-07, from the conversation's post-mortem).
|
||||
- **A2 — the overview tail**: `Sources: <name> (<k> doc|docs), …` appended server-side to the stored outline at generation time (happy path only); deterministic, non-LLM, catalog order; `build_overview_prompt` / `OVERVIEW_INSTRUCTION` / `load_kb_overview` / `build_kb_section` byte-identical; the zero-doc and `LLMError` paths untouched (owner-confirmed 2026-10-07).
|
||||
- **A3 — the grep fold**: `grep_document` compares the `_fold`+lowercase of the pattern against the `_fold`+lowercase of each line — the SAME `_fold` as A1 (imported); still a fixed substring (phase-68 lock), still a locator, regex teaching byte-identical (owner-confirmed 2026-10-07 — the conversation's "optional grep-side accent folding", accepted).
|
||||
- **A4 — the D4 sentence** is appended to the `TOOLS_SECTION` tools paragraph immediately after the grep sentence, verbatim as written in D4; `PERSONA` + LOW body + the other `TOOLS_SECTION` sentences byte-identical (owner-confirmed 2026-10-07 — the owner asked for the system-prompt improvement in the conversation).
|
||||
- **A5 — PLAN.md untouched (Protocol B)**: the grep-contract extension (A3) and the prompt sentence (A4) are recorded here (owner-confirmed) and folded into the anchor table at the next redo (the phase-119 A6 precedent).
|
||||
- **A6 — the steering-note follow-up is an owner runtime action, not code**: the conversation's proposed tuning rule ("IKEA's BESTÅ assembly PDF is a live source — search it as BESTÅ or AA-1272068, not besta") is a `steering_notes` row the owner adds via the Tuning tab after this phase ships; the phase records it here and does not touch `steering_notes` data.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): accent-fold name hits and grep (BESTÅ <-> besta), a deterministic source inventory in the KB overview, and search-discipline prompt teaching"
|
||||
```
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 01 — The accent-folded name normalization (phase-162 D1, LOCKED A1)
|
||||
|
||||
**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D1 — the 2026-10-07 live finding: `grep "besta"` never hit the IKEA PDF (its text layer carries only `BESTÅ`, U+00C5), and the symmetric name-hit gap: the question token `BESTÅ` normalizes (lower + alnum-only) to `best`, never `besta`, so an accented question cannot name-match the ASCII filename `besta-frame-white__AA-1272068-11-100` (whose stem sub-components include `besta` under phase-119 LOCKED A2).
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`_normalize_name` in `app/rag/retriever.py` folds accents (NFKD + combining-class strip, stdlib `unicodedata`) before the existing alnum scan, so both sides of name matching bridge `BESTÅ` ↔ `besta` — with the A2 mechanism (match classes, ranking, caps, flags, bonus) and the FTS/cosine/A8 pipeline byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/retriever.py` — new module-level `_fold(s: str) -> str`: `unicodedata.normalize("NFKD", s)` with every char whose `unicodedata.combining(c)` is non-zero dropped (stdlib only — no new dependency). Docstring: the shared accent fold (phase 162) — NFKD decomposition, combining marks dropped; `Å/å → a`, `ß → ss`, `fi → fi`, `ç → c`; characters with no decomposition (CJK, `①`) survive the fold and still drop out under the caller's alnum scan (behavior unchanged for them).
|
||||
2. `app/rag/retriever.py` — `_normalize_name`: `"".join(_ALNUM_RE.findall(_fold(s.lower())))` (was: `…findall(s.lower())`). Docstring update: the folded form (`Qwen 3.8` → `qwen38` unchanged; `BESTÅ` → `besta` new) — this is the **single** normalization both sides pass through (the `name_hit_tokens` question tokens and the `_name_parts` path parts / stem / stem sub-components), so the one change bridges both directions.
|
||||
3. `app/rag/retriever.py` — `name_hit_tokens` / `_name_parts` / `_name_token_matches`: **no code change** (they consume `_normalize_name`); their docstrings gain one clause each: candidates/parts are accent-folded (phase 162) — `besta` matches `BESTÅ`-class names both ways.
|
||||
4. `app/rag/retriever.py` — module docstring: the normalization paragraph now states the fold (the 2026-10-07 gap: the ASCII filename vs the accented content layer).
|
||||
5. `tests/unit/test_retriever.py` — the fold, on the file's existing fixture pattern:
|
||||
- **question side**: the accented question "How do I build the BESTÅ?" produces a `besta` token (was `best` pre-phase — pin the exact token list); `Ä`/`Ö`/`Ü`/`ß`/`fi`/`ç` fold cases (`straße`-class → `strasse`-class, ligature `fine` → `fine`);
|
||||
- **path side**: `_name_parts` on `besta-frame-white__AA-1272068-11-100` yields `besta` in the equal set (the phase-119 case, unchanged) AND on the accented filename `BÅsta.md` yields `basta` — the ASCII question `basta` name-matches it;
|
||||
- **the bridge, end to end**: `_name_hit_chunks`-class assertion — an accented-question token hits the ASCII-stem doc AND an ASCII token hits the accented-stem doc;
|
||||
- **A2 non-regression (all existing behavior, re-pinned after the change)**: digitless precision (the `deploy`-titled `Deployments/` doc does NOT hit for `deploy`), digit-bearing prefix (`qwen38` → `qwen3.8-27b-…`, and the mid-word non-match), the versioned join (`Qwen 3.8` → `qwen38`), min-length on the **folded** form, ranking (matched-token count, catalog order), `NAME_HIT_LIMIT`, `name_hit=True`/`False` convention;
|
||||
- **no-decomposition characters**: a CJK / `①` token still drops out (the alnum scan behavior for non-folding chars is unchanged);
|
||||
- the FTS/`retrieve()`/gate rows are untouched (the existing `test_retriever.py` / `test_retriever_recency.py` suite stays green without modification).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `BESTÅ` (question) → `besta` token → name-hits the ASCII filename's sub-components, and `besta` (question) → name-hits the accented filename `BÅsta.md` — one function, both directions.
|
||||
- [ ] Every phase-119 A2 case (precision, prefix, join, ranking, cap, flag) passes UNCHANGED after the fold; the FTS/gate suites are untouched.
|
||||
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_retriever_recency.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
# Task 02 — The deterministic source inventory in the KB overview (phase-162 D2, LOCKED A2)
|
||||
|
||||
**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D2 — the 2026-10-07 live finding: the `lite`-generated `<knowledge_base>` outline listed 3 groups of a 14-source index; the 1-doc sources `ikea_instructions` and `players_handbook` were invisible, so source-level questions ("You have ikea_instructions in your RAG" — the owner's push-back) could not be surfaced by the outline at all.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`regenerate_overview` in `app/rag/overview.py` appends one **deterministic, non-LLM** line to the stored outline — `Sources: <name> (<k> doc|docs), …` — so no source can be invisible to `<knowledge_base>`; the model input, the load/inject path, and the zero-doc / LLM-failure paths are byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/overview.py` — a new pure helper `_source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str` next to `_first_summary_line`: one entry per **distinct** source, in catalog order (first occurrence in the `source, path`-ordered rows), each `<name> (<k> doc|docs)` — singular `doc` at k=1, `docs` otherwise — joined with `, ` behind the prefix `Sources: `. (The `rows` shape is the existing `(source, path, title, summary)` projection `regenerate_overview` already reads — no new query.) Docstring: the phase-162 deterministic inventory (LOCKED A2) — mechanical, non-LLM; the `lite` outline stays the model's job.
|
||||
2. `app/rag/overview.py` — `regenerate_overview` **happy path only**: after the model text is obtained and before the upsert, the stored content = the model text (trailing newlines rstripped) + `"\n"` + the inventory line. The upsert, the `overview: regenerated docs=… chars=…` log (chars = the FULL stored content, tail included), the `True` return, and the change-gating contract are otherwise unchanged.
|
||||
3. `app/rag/overview.py` — the **zero-doc early return** (`False`, row untouched) and the **`LLMError` path** (logged, `False`, previous row intact) are UNCHANGED — the previous row keeps its earlier tail, it is part of the stored content; an old outline is better than none. `build_overview_prompt`, `OVERVIEW_INSTRUCTION`, `SYSTEM_PROMPT`, `load_kb_overview`, and `app/rag/prompts.py::build_kb_section` are **untouched** (the model never sees the tail; the existing longest-prefix cap applies to the whole stored text — at the default `BOR_KB_OVERVIEW_MAX_CHARS` (4000) the ~1500-char outline + the one-line tail always fit; a smaller owner-set cap cuts the tail first — accepted, recorded in the docstring).
|
||||
4. `app/rag/overview.py` — module docstring: a paragraph on the inventory line (phase 162 — the 2026-10-07 finding: the `lite` outline is lossy; small sources must be visible without a model call; the line is deterministic so the mock/real model outputs never affect it).
|
||||
5. `tests/unit/test_overview.py` — the helper + the paths (the file's existing duck-typed-`OverviewLLM` fake pattern):
|
||||
- the format: `Sources: a (1 doc), b (2 docs), c (3 docs)` — exact, incl. singular/plural and the catalog order;
|
||||
- the **1-doc survival** (the `ikea_instructions` case): a 1-doc source among many appears with `(1 doc)`;
|
||||
- the happy path: the stored row = model text + newline + the tail (exact content pin on the fake LLM's fixed output);
|
||||
- zero-doc: `False`, the existing row (a pre-phase row, tail-less) byte-identical;
|
||||
- `LLMError`: `False`, the **previous** row byte-identical — INCLUDING a row that already carried a tail (the tail is content, preserved on failure);
|
||||
- `build_overview_prompt` byte-identical to the pre-phase output for the same rows (the model input never gains the tail);
|
||||
- the regenerated log line carries the full stored-content char count.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A regenerated `kb_overview` row ends with `Sources: …` naming every source with exact counts (a 1-doc source visible); the model input is byte-identical to the pre-phase prompt.
|
||||
- [ ] The zero-doc and `LLMError` paths leave the row byte-identical (an existing tail preserved on failure); the char log is exact.
|
||||
- [ ] `uv run pytest tests/unit/test_overview.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
# Task 03 — The accent-folding grep + the search-discipline prompt teaching (phase-162 D3+D4, LOCKED A3+A4)
|
||||
|
||||
**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D3+D4 — the 2026-10-07 live finding: `grep "besta"` never hit the PDF whose text layer carries only `BESTÅ` ("the tool is documented as 'a plain substring' and silently missing Å is the nastiest of the four"), and the owner's ask: "Let's try to improve your search techniques and system prompt."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`grep_document` folds accents on **both** pattern and content (the SAME `_fold` as task 01, imported — grep and the name-hit path can never drift); the contract copy in all three places states the folded contract; the LOCKED D4 search-discipline sentence joins the `TOOLS_SECTION` tools paragraph; the phase-117 pins re-cut in this task.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `grep_document(content, pattern)`: the comparison is `_fold(pattern.lower()) in _fold(line.lower())` per (rstripped) line — importing `_fold` from `app.rag.retriever` (the module already imports `TRUNCATION_MARKER` from there). Everything else UNCHANGED: the rstrip, the `lineno` numbering, the 20-match cap, the 200-char line truncation, the scoped/unscoped call sites (`all_documents` / the one-doc path at the existing call site), the locator semantics (no context add — `holder.read_docs` untouched). Docstring update: case- AND accent-insensitive fixed substring (phase 162, LOCKED A3) — the fold changes the alphabet it ignores, not the matching semantics; still NEVER a regex.
|
||||
2. `app/rag/agent.py` — the module docstring grep paragraph (the "case-insensitive fixed substring" sentence): "case- and accent-insensitive fixed substring (accents fold — `besta` matches `BESTÅ`)".
|
||||
3. `app/rag/agent.py` — `AGENT_TOOLS` the grep function description: "(case-insensitive)" → "(case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)", and the `pattern` parameter description gains the same clause (the "plain substring, NEVER a regex" teaching unchanged). The `NO_MATCHES_REGEX` / `NO_MATCHES_REGEX_SCOPED` teaching lines and `looks_like_regex` / `plain_form` are **byte-identical** (the plain-substring claim they teach is still true).
|
||||
4. `app/rag/prompts.py` — `TOOLS_SECTION`, the grep sentence: "locates an exact string (case-insensitive)" → "locates an exact string (case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)" (the sentence's remainder, incl. the plain-substring + retry teaching, unchanged).
|
||||
5. `app/rag/prompts.py` — `TOOLS_SECTION`: the LOCKED D4 sentence (verbatim from the overview) appended to the tools paragraph **immediately after the grep sentence**, i.e. after the `…retry with the plain text you expect to see.` clause:
|
||||
`A summary or outline is a hint, never an inventory: before claiming the knowledge base lacks something, run `ls` with no `path` to list every source, and `grep` the artifact's part number or a filename token. `grep` folds accents — `besta` matches `BESTÅ` — but one pattern is one string: a name's split tokens (`frame`, `white`) and its part number (`AA-1272068`) need separate greps. A note saying a file is gitignored describes its repo status, not whether it is indexed as a source.`
|
||||
`PERSONA`, the LOW/`DEFLECT_MODE` body, the `<documents>` blocks, and every other `TOOLS_SECTION` sentence are byte-identical. (The E2E mock is unaffected — it keys on markers, not on the sentence.)
|
||||
6. Pins re-cut, **in this task** (the phase-124 pattern): `tests/unit/test_prompt_lock.py` (the `TOOLS_SECTION` pin + its derived cross-pins), `tests/unit/test_prompts.py` (the grounded-prompt pins that embed the grep sentence), `tests/unit/test_agent.py` (the `AGENT_TOOLS` description pin, if present — grep the file for the old description text; the `NO_MATCHES_REGEX` pins stay green unmodified).
|
||||
7. `tests/unit/test_agent.py` — `grep_document` fold coverage (the existing case-insensitive test's pattern): `grep_document("The BESTÅ frame (AA-1272068-11-100)", "besta")` hits; `grep_document("…BÅsta…", "basta")` hits; `grep_document("café", "cafe")` hits; the pattern's own accent folds too — `grep_document("The besta frame", "BESTÅ")` ALSO hits (the pattern `BESTÅ` folds to `besta`); a genuinely non-matching folded pattern still returns no lines (`grep_document("The besta frame", "bestb")` → `[]`); line numbers / rstrip / cap / truncation unchanged (existing tests stay green unmodified).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `grep_document` matches accented ↔ ASCII both ways through the imported `_fold` (no second copy of the fold); the regex teaching lines are byte-identical.
|
||||
- [ ] The three copy sites (module docstring, `AGENT_TOOLS`, `TOOLS_SECTION`) state the folded contract; the D4 sentence sits immediately after the grep sentence, LOCKED verbatim.
|
||||
- [ ] The re-cut `test_prompt_lock.py` / `test_prompts.py` / `test_agent.py` pins green; the LOW + `PERSONA` byte-identity pins green unmodified.
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — The dedicated E2E, the measured pre-fix evidence, and the battery pin (phase-162 D5)
|
||||
|
||||
**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D5 — the 2026-10-07 finding must turn from anecdote into a regression test: the accented question "How do I build the BESTÅ?" surfaces the accented-content doc whose ASCII filename carries the name's sub-components, over ASCII "besta"-mentioning distractors; the 1-doc source is visible in the stored overview; the A8 gate and the five phase-119 battery questions do not move.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`tests/e2e/test_search_recall_small_sources.py` (run in isolation) gates the fold end-to-end on a synthetic replica of the live miss; the pre-fix ranks are **measured and recorded** in this file (the under-rank hypothesis is pinned green or handed to the owner with evidence — never assumed); `tests/fixtures/retrieval_battery.txt` gains the Besta question per D5; the final gates run.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_search_recall_small_sources.py` — new dedicated suite, **run in isolation** (`uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov`), on the phase-119 local-directory-source fixture pattern (fixture module truncates/re-imports; the deterministic mock LLM by default). Fixture KB — a synthetic replica of the live miss:
|
||||
- source **`ikea_instructions`** (ONE doc): `besta-frame-white__AA-1272068-11-100.md` — body carrying the product name **only in accented form** (e.g. "The BESTÅ frame (AA-1272068-11-100)… cam bolt 130618 … wall bracket 141413 … leveling foot …") — **no ASCII "besta" anywhere in the body** (the live PDF's exact situation);
|
||||
- 2–3 **distractor docs** in other sources whose bodies contain ASCII "besta" (the phase-docs stand-in — e.g. `brain-of-reese/.agents/phases/complete/140_pictorial_pdf_pages/00_phase.md`-shaped text: "the besta fixture drove the pictorial phases"). Tune the distractor count/bodies so the pre-phase accented question loses the `ikea_instructions` doc from the suggested top-5 (step 2 measures this — if it does NOT lose it, add distractors until it does, and record the final fixture shape here).
|
||||
Assertions (mock LLM; the `done`-frame `suggestions` list is the observable tier — the phase-119 done-frame shape):
|
||||
- **(1) the accented question** `POST /api/chat` "How do I build the BESTÅ?" → the `done` frame's `suggestions` include `ikea_instructions/besta-frame-white__AA-1272068-11-100.md` (the D1 question-side fold — `BESTÅ` → `besta` → the stem sub-component name-hit);
|
||||
- **(2) the ASCII regression** "How do I build a Besta?" → the same doc in `suggestions` (the phase-119 path, unchanged by the fold);
|
||||
- **(3) the overview tail** — after the fixture import (the mock's `KB_OVERVIEW_MODE` regeneration ran), the stored `kb_overview` row (read through the fixture's DB session, the phase-119 pattern) ends with `Sources: …` naming **every** fixture source with exact counts — the 1-doc `ikea_instructions` source visible (the D2 gate);
|
||||
- **(4) the gate sanity** — an out-of-KB question (e.g. "What is the capital of Mongolia?") returns the deflected terminal (the D1–D3 changes must not move the A8 gate verdict; the mock's cosine behavior makes this deterministic — follow the phase-119 deflection-sanity assertion).
|
||||
2. **The pre-fix evidence — measured, recorded in THIS file (before the final gates):** `git stash` the task-01 `_normalize_name` change (tasks 02–03 changes may stay — they do not affect the name-hit path), run assertion (1) against the fixture, record the measured suggested top-5 ranks for the accented question (the `ikea_instructions` doc's rank or its absence), `git stash pop`. The recorded block must show the doc OUT of the suggested top-5 pre-fix (if the fixture as built does not show that, go back to step 1's tuning and re-measure — the recorded block is the evidence the fold fixes a real miss, not a no-op).
|
||||
3. `tests/fixtures/retrieval_battery.txt` — append `How do I build a Besta?` with a gate comment (the file's existing comment style): on the 1,000-doc live replica (the `restore-test-db` skill — the dump is a local gitignored artifact; **if missing, STOP and ask the owner, never fabricate numbers**) the battery runs when `SELECT DISTINCT source FROM documents` shows an `ikea_instructions`-class source, and the printed `suggested:` top-5 must include `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`; if the replica lacks that source (the snapshot may predate the import), the five EXISTING phase-119 questions still run — their recorded gate (the canonical docs' ranks, the qwen3.8 regression, the deflection sanity `HIGH, fts_hits=5`) must be **unchanged** by this phase — and the Besta question is recorded in the task file as **skipped loudly (source absent in the replica snapshot)** — the phase-140 "skipped loudly" precedent. The under-rank hypothesis (the overview's failure #4) is thus either pinned green or handed to the owner with the measured ranks.
|
||||
4. Regression + final gates: the phase-119 (`test_name_signal_read_chips.py`), phase-117, and phase-31-era regression suites green; `uv run pytest` (full suite) green; `uv run pytest --cov=app --cov-report=term-missing` coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the dedicated suite in isolation (per AGENTS.md rule 4 — one file per phase, run in isolation); unit pins from tasks 01–03 already in place.
|
||||
- Battery: per step 3, recorded in this file.
|
||||
- Coverage: **>90%** on `app/`; lint + types clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The recorded pre-fix block in this file shows the accented question losing the `ikea_instructions` doc from the suggested top-5; the post-fix E2E (assertions 1–4) is green in isolation.
|
||||
- [ ] The battery question is recorded per step 3 (pinned green, or skipped loudly with the five phase-119 questions' gate verified unchanged).
|
||||
- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] The owner follow-up (A6) is noted in the final pass's summary for the owner (the steering-note "rule 13" is a runtime Tuning-tab action, not code).
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**
|
||||
|
||||
- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
|
||||
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
|
||||
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
|
||||
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
|
||||
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
|
||||
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
|
||||
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
|
||||
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
|
||||
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
|
||||
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
|
||||
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
|
||||
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
|
||||
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
............ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 764 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 470 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5788 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
All gates green. Final verification complete.
|
||||
|
||||
**Report**
|
||||
|
||||
- **Implemented (task 162/01, D1 / LOCKED A1):**
|
||||
- `app/rag/retriever.py`: new module-level `_fold()` (stdlib NFKD + combining-class strip, shared with task 03's grep); `_normalize_name` = alnum scan of `_fold(s.lower())` — the single normalization both sides pass through; `name_hit_tokens`/`_name_parts`/`_name_token_matches` consumed it unchanged (docstring clauses only); module docstring normalization paragraph states the 2026-10-07 gap.
|
||||
- `tests/unit/test_retriever.py`: +9 tests — `_fold` cases (Å/ÄÖÜ/ß/fi/ç, CJK, ①), `BESTÅ`→`besta` token pinned exactly (`["build", "besta"]`), `_name_parts` (`besta` in equal set unchanged; `BÅsta.md`→`basta`), end-to-end bridge both directions via `_name_hit_chunks`, min-length on the folded form, digitless-precision non-regression (`déploy` ≠ `Deployments`).
|
||||
- **Tests/lint/coverage:** `uv run pytest tests/unit/test_retriever.py tests/unit/test_retriever_recency.py -v` → 94 passed; full `uv run pytest` → **3747 passed**; `--cov=app` → 99% TOTAL, retriever 99% (3 pre-existing misses); `uv run ruff check .` + `uv run pyright` clean; phase-119 E2E `tests/e2e/test_name_signal_read_chips.py` → 4 passed (FTS/gate suites untouched).
|
||||
- **Docs:** README "How retrieval works" — lexical list now names the name-hit path + accent fold (phase 162). No AGENTS.md impact (its grep-contract line is task 03/final-pass scope).
|
||||
- **Decisions/deviations:**
|
||||
1. NFKD alone leaves `ß` intact; the task-pinned `ß → ss` case requires `str.casefold()` — applied inside `_fold` (still stdlib-only, single shared function, A1 intent intact; documented).
|
||||
2. `①` is compat-decomposable (NFKD → `1`), contrary to the task's "no decomposition" note: it still drops as a token (<4 chars); its only footprint is a prefix-class versioned-join artifact (`the1`) no path can match — pinned as real behavior and documented.
|
||||
- **Next pending task:** `162_search_recall_small_sources/02_kb_overview_source_inventory.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 764 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 470 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5780 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
All gates are green. Task 02 is complete.
|
||||
|
||||
**Report — task 02 `02_kb_overview_source_inventory.md`**
|
||||
|
||||
- **Implemented** (verified/completed in working tree — a prior interrupted run had left the code/tests uncommitted; I audited them against D2/A2, confirmed byte-identity of `build_overview_prompt`/`OVERVIEW_INSTRUCTION`/`load_kb_overview`/`build_kb_section`, and finished the doc work):
|
||||
- `_source_inventory(rows)` in `app/rag/overview.py` — deterministic, non-LLM `Sources: <name> (<k> doc|docs), …`, distinct sources, catalog order, singular/plural.
|
||||
- `regenerate_overview` happy path stores model text (rstripped `\n`) + `\n` + tail; log `chars=` = full stored content; zero-doc and `LLMError` paths untouched (previous row + tail preserved).
|
||||
- Module + function docstrings record phase 162 / A2 / the `BOR_KB_OVERVIEW_MAX_CHARS` budget note.
|
||||
- Unit tests in `tests/unit/test_overview.py`: format/order/counts, 1-doc survival (`ikea_instructions` case), exact happy-path stored content, tail-less zero-doc row preserved, `LLMError` preserves an existing tail, model input never gains the tail, exact char log; re-cut pins in `tests/integration/test_import_docs_overview.py`, `tests/e2e/test_kb_overview.py`, `tests/e2e/test_kb_overview_freshness.py`.
|
||||
- **Tests / lint / coverage** (DB container started — it was down):
|
||||
- `uv run pytest tests/unit/test_overview.py -v` → 29 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **3753 passed**, `app/` TOTAL **99%**, `app/rag/overview.py` **100%**
|
||||
- `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` → 4 passed; `tests/e2e/test_kb_overview_freshness.py` → 3 passed
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||
- **Docs**: `README.md` — import section now documents the deterministic `Sources: …` tail (phase 162) and the failure-path preservation; `AGENTS.md` — no doc impact (the grep-contract line there is task 03/04's scope).
|
||||
- **Decisions**: inventory order via `sorted()` on source names = catalog order regardless of the recency-first row order the generator reads (proven by the scrambled-input test); no other e2e asserts exact overview-row content (checked).
|
||||
- **Next pending task**: `03_grep_folding_search_discipline.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
......... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 764 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 470 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5787 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
Task 03 complete. Report:
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/agent.py` — `grep_document` now folds **both** pattern and each line through the shared `_fold` imported from `app.rag.retriever` (one fold, no second copy; LOCKED A3): `_fold(pattern.lower()) in _fold(line.lower())`; rstrip/line-numbers/20-cap/200-char truncation/locator semantics untouched.
|
||||
- Copy updates (LOCKED A3): module-docstring grep paragraph, `AGENT_TOOLS` grep function description, and `pattern` parameter description now state "case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`"; `NO_MATCHES_REGEX`/`_SCOPED` + `looks_like_regex`/`plain_form` byte-identical.
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` grep sentence folded-contract re-worded; new `SEARCH_DISCIPLINE_RULE` constant (LOCKED A4 sentence, verbatim) inserted immediately after the grep sentence; `PERSONA`/LOW/other sentences byte-identical.
|
||||
- Pins re-cut (phase-124 pattern): `test_prompt_lock.py` (new sha `988642f9…`/len 3818, pre-phase-162 anchors, chained phase-162→145 diff-style pin, discipline-rule position pin, grep-contract pin), `test_prompts.py` (grep clause + D4 boundary pins), `test_agent.py` (`AGENT_TOOLS` + pattern-param description pins).
|
||||
- New fold tests in `test_agent.py`: accented↔ASCII both ways (`besta`→`BESTÅ`, `basta`→`BÅsta`, `cafe`→`café`, `BESTÅ` pattern→ASCII content, `strasse`→`Straße`), non-match still `[]`.
|
||||
- `README.md` — grep tool table row updated to the folded contract.
|
||||
|
||||
**Tests / lint / coverage**
|
||||
- `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py` → **292 passed**
|
||||
- `uv run pytest` (full suite) → **3756 passed**
|
||||
- Related E2E spot-checks: `test_grep_regex_teaching.py`, `test_tool_call_compact.py`, `test_kb_overview*.py`, `test_name_signal_read_chips.py` → **green** (11 passed in the last batch)
|
||||
- `uv run pytest --cov=app` → TOTAL **99%** (`agent.py` 99%, `prompts.py` 100%) — >90% gate met
|
||||
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
|
||||
|
||||
**Docs**: `README.md` grep-tool row updated (case- and accent-insensitive, fold example); `AGENTS.md` has no grep-contract line to change (the phase-162 final pass handles the §6 wording per the phase overview).
|
||||
|
||||
**Decisions**: reworked the phase-145 "only byte change" pin into a chained phase-162→145 revert pin so both historical anchors stay enforced; `SEARCH_DISCIPLINE_RULE` added as a module constant mirroring the `ORIGIN_RULE` pattern. No deviations from LOCKED A3/A4.
|
||||
|
||||
**Next pending task**: `162_search_recall_small_sources/04_e2e_battery_pins.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
............ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 764 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 470 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5788 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
**Phase 162 / task 04 — complete, all gates green.**
|
||||
|
||||
**Added** `tests/e2e/test_search_recall_small_sources.py` (7 tests), fixture KB `tests/fixtures/recallkb/` (3 sources: `ikea_instructions` 1 doc with the name **only** as `BESTÅ`, `phase_docs` 4 + `players_handbook` 1 ASCII-`besta`/`build` distractors), the Besta pair + gate comment in `tests/fixtures/retrieval_battery.txt`, and evidence in `.agents/reports/162_search_recall_small_sources/task04_prefix_evidence_and_battery.md`.
|
||||
|
||||
**Commands / outcomes**
|
||||
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated, run twice incl. after the DB restore).
|
||||
- `uv run pytest tests/e2e/test_name_signal_read_chips.py tests/e2e/test_kb_overview.py -q --no-cov` → **8 passed**.
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, 3756 tests, app coverage 99%**.
|
||||
- `uv run ruff check .` → passed; `uv run pyright` → **0 errors**.
|
||||
- Replica restored per the `restore-test-db` skill (fingerprint verified docs=1000/chunks=8866/sources=deploy,homelab,homepage,ServMon,skills,vibe-bot, alembic 0011→0026), then `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` → **GATE PASS 1–5**: gitea README rank 4, forgejo 5/5 with README #1, gateway README rank 3, qwen3.8-27b quadlets #1/#2, Mongolia HIGH/fts_hits=5 — the five `suggested:` lists are **byte-identical** to the phase-119 record, and a re-run with the fold reverted is **byte-identical** to the post-fix run.
|
||||
|
||||
**Pre-fix evidence (measured, both directions).** With `app/rag/retriever.py::_normalize_name` reverted by a one-line reversible edit, `How do I build the BESTÅ?` loses the doc: tokens `['build','best']` → no name hit, suggested 5 = the five ASCII distractors, IKEA doc **rank 6** — visible on the wire as its `done.related` entry and its absence from the folder-map echo (gate still HIGH, cosine 0.382, on the wrong docs). Shipped code: tokens `['build','besta']` → IKEA doc **suggested #1**, related = `phase_docs/156_history_source_filter.md`, cosine unchanged (0.382). The hypothesis is now re-measured on every run by `test_under_rank_is_measured_not_assumed`.
|
||||
|
||||
**Decisions / deviations** (harness mechanics win): task/phase files untouched, so the evidence lives in the report file above; no `git stash` (reversible source revert instead, verified restored, `grep -c TEMP` = 0); `done.suggestions` is the deflection-chip contract and never carries paths, so the suggested tier is asserted via the durable `query_log.sources` record (suggested 5 + remainder), `done.related`, and the mock's folder-map echo; tuning re-worded two bodies to drop stopword density so the out-of-KB control deflects (cosine 0.3345 → 0.2083) — both token controls are pinned as assertions; the battery carries both `How do I build a Besta?` (task text) and `How do I build the BESTÅ?` (the live miss).
|
||||
|
||||
**Skipped loudly:** the 2026-09-05 replica snapshot has no `ikea_instructions`-class source (`grep -c -i ikea` on the dump = 0), so the Besta rank gate stays with the owner's live KB; both questions still ran without a false grounding on a doc naming the product.
|
||||
|
||||
**Docs:** no README/AGENTS change needed (task 04 is tests + fixtures + battery; user-visible behavior was documented by tasks 01–03). DB left as found (pre-restore backup restored); no other pending task for this phase — task 04's file is ready for the harness to move, and the phase commit is still owed.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
............ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 764 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 470 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5788 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
# Task 04 record — pre-fix evidence + battery (phase 162, D5/D6)
|
||||
|
||||
**Deviation note (mechanics):** the task file says "record the measured
|
||||
pre-fix ranks in THIS file" and to revert the fold with `git stash`, but
|
||||
the execution harness forbids editing phase/task files and forbids
|
||||
index/history-touching Git commands. Per the conflict rule the harness
|
||||
wins on mechanics and the intent is preserved: the evidence lives HERE,
|
||||
the fold was reverted by a one-line, reversible edit of
|
||||
`app/rag/retriever.py` (restored and byte-verified afterwards —
|
||||
`grep -c TEMP app/rag/retriever.py` = 0), and the summary goes in the
|
||||
task's final report. The permanent, non-throwaway form of the evidence is
|
||||
`tests/e2e/test_search_recall_small_sources.py::test_under_rank_is_measured_not_assumed`,
|
||||
which re-measures BOTH directions on every run.
|
||||
|
||||
## 1. The fixture (tests/fixtures/recallkb — three local-directory sources)
|
||||
|
||||
| source | docs | role |
|
||||
| --- | --- | --- |
|
||||
| `ikea_instructions` | **1** (`besta-frame-white__AA-1272068-11-100.md`) | the live miss: body names the product **only** as accented `BESTÅ` |
|
||||
| `phase_docs` | 4 (`140_pictorial_pdf_pages`, `148_pdf_text_backfill`, `152_seaweedfs_blob_ingest`, `156_history_source_filter`) | ASCII `besta` + `build` distractors |
|
||||
| `players_handbook` | 1 (`game_night_setup.md`) | ASCII `besta` + `build` distractor in a second small source |
|
||||
|
||||
Final shape tuning (the task's "add distractors until it loses"): the
|
||||
initial 4-doc draft already lost the tier, but the deflection control
|
||||
failed (best cosine 0.3345 — the small source's `Assembly` section was
|
||||
dense in the stopword `the`, which the mock's tokenizer keeps). Two
|
||||
bodies were re-worded (`ikea_instructions` Assembly paragraph,
|
||||
`players_handbook/game_night_setup.md`) to drop stopword density: the
|
||||
out-of-KB cosine fell to **0.2083** (< 0.30 threshold → deflects) while
|
||||
the on-topic questions kept 0.38 / 0.44. Two token controls are pinned
|
||||
as assertions in `synced_kb`: (a) no ASCII `besta` and no `build` anywhere
|
||||
in the small source, (b) the accented name sits **past** the mock's
|
||||
24-token summary-digest window, so the ASCII-only digest never leaks a
|
||||
`best` lexeme the pre-phase FTS could have hit.
|
||||
|
||||
## 2. Measured pre-fix evidence — the E2E fixture, both directions
|
||||
|
||||
Lens: the app's REAL `retrieve()` + `select_suggested()` over the synced
|
||||
KB with the mock's embeddings. Gate: threshold 0.30, lexical-support
|
||||
floor 0.15 (the conftest mock calibration). Bonus 0.005, recency 0.0007,
|
||||
suggested 5 / related 2 (code defaults, `.env`-free like production).
|
||||
|
||||
Question `How do I build the BESTÅ?`
|
||||
|
||||
**PRE-FIX — the real revert** (`_normalize_name` →
|
||||
`"".join(_ALNUM_RE.findall(s.lower()))`, the exact task-01 before-state),
|
||||
`pytest tests/e2e/test_search_recall_small_sources.py` run against the
|
||||
fixture (full log: `/tmp/prefix_e2e_run.log`, `/tmp/prefix_wire_run.log`):
|
||||
|
||||
```
|
||||
name tokens : ['build', 'best'] → NO name hit
|
||||
suggested 5 : phase_docs/140_pictorial_pdf_pages.md
|
||||
phase_docs/148_pdf_text_backfill.md
|
||||
phase_docs/152_seaweedfs_blob_ingest.md
|
||||
players_handbook/game_night_setup.md
|
||||
phase_docs/156_history_source_filter.md
|
||||
related : ikea_instructions/besta-frame-white__AA-1272068-11-100.md ← RANK 6, the miss
|
||||
gate : best cosine 0.382, fts_hits 8 → grounded (on the WRONG docs)
|
||||
done frame : {'source': 'ikea_instructions',
|
||||
'path': 'besta-frame-white__AA-1272068-11-100.md',
|
||||
'title': 'Assembly Instructions'} ← the rank-6+ "Nearby docs" slot
|
||||
folder echo : (folders: phase_docs/: 148_…, 152_…, 156_…; players_handbook/:)
|
||||
→ NO `ikea_instructions/` line at all: the small source never
|
||||
reached the HIGH prompt
|
||||
```
|
||||
|
||||
**POST-FIX (shipped code)** — same fixture, same measurement:
|
||||
|
||||
```
|
||||
name tokens : ['build', 'besta'] → name hit
|
||||
suggested 5 : ikea_instructions/besta-frame-white__AA-1272068-11-100.md ← #1
|
||||
phase_docs/140_pictorial_pdf_pages.md
|
||||
phase_docs/148_pdf_text_backfill.md
|
||||
phase_docs/152_seaweedfs_blob_ingest.md
|
||||
players_handbook/game_night_setup.md
|
||||
related : phase_docs/156_history_source_filter.md
|
||||
gate : best cosine 0.382, fts_hits 9 (the fold adds the LEXICAL term;
|
||||
cosine untouched — LOCKED A1)
|
||||
folder echo : … includes `ikea_instructions/:` (header alone — its only
|
||||
document IS the suggested doc)
|
||||
```
|
||||
|
||||
Mechanism (measured, and why the fix is the name signal and nothing
|
||||
else): every distractor matches the question's other FTS token `build`,
|
||||
so each fuses TWO RRF terms (vector + FTS ≈ 0.032) while the small source
|
||||
— no `build`, and `BESTÅ` lexes as `bestå`, which the ASCII-`best`
|
||||
tsquery cannot match — fuses ONE (≈ 0.016) and lands rank 6. The folded
|
||||
name hit is the only signal big enough to lift it: name hits LEAD the
|
||||
lexical list (`fuse` rank 1 = 1/61) plus the per-document
|
||||
`BOR_NAME_HIT_BONUS` (0.005). ASCII question `How do I build a Besta?`
|
||||
measures the SAME tier before and after (suggested #1, cosine 0.4404,
|
||||
related `phase_docs/156_history_source_filter.md`) — the phase-119 path
|
||||
is untouched.
|
||||
|
||||
Under-rank hypothesis (D1 failure #4): **pinned green with measured
|
||||
ranks**, not assumed.
|
||||
|
||||
## 3. The dedicated suite
|
||||
|
||||
`uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov`
|
||||
→ **7 passed** (task-04 cases 1–4 plus the measurement test, the grep
|
||||
fold at the content layer, and the folder-line prompt witness).
|
||||
|
||||
Interpretation note: the task text says the `done`-frame `suggestions`
|
||||
list is the observable tier. On the shipped wire `done.suggestions` is
|
||||
the DEFLECTION contract (2–3 title-derived "Maybe try" chips, `[]` on
|
||||
grounded turns — `app/api/chat.py`, phase 04) and never carries document
|
||||
paths. The faithful observables of the SUGGESTED tier are (a) the durable
|
||||
`query_log.sources` record = suggested 5 then the rank-6+ remainder
|
||||
(phase 118 A3) and (b) `done.related` = the rank-6+ tier. The suite
|
||||
asserts both (plus `done.sources == []` — read-docs-only, phase 119
|
||||
LOCKED A1 — and the mock's folder-map echo for the prompt-level witness).
|
||||
|
||||
Neighbour suites re-run (the retriever + overview were touched by this
|
||||
phase): `uv run pytest tests/e2e/test_name_signal_read_chips.py
|
||||
tests/e2e/test_kb_overview.py -q --no-cov` → **8 passed**.
|
||||
|
||||
## 4. Battery — `tests/fixtures/retrieval_battery.txt`
|
||||
|
||||
The file gains the Besta pair (`How do I build a Besta?` — the task's
|
||||
line — and `How do I build the BESTÅ?`, the live miss itself) with the
|
||||
gate comment. The five phase-119 questions keep positions 1–5.
|
||||
|
||||
Preconditions (the `restore-test-db` skill, run in-container): safety
|
||||
backup → `DROP SCHEMA public CASCADE` → single-transaction load of
|
||||
`data/bor_live_kb_replica.dump.sql` → **fingerprint verified:
|
||||
docs=1000, chunks=8866,
|
||||
sources=deploy,homelab,homepage,ServMon,skills,vibe-bot, alembic=0011**
|
||||
→ `uv run alembic upgrade head` (0011 → **0026**) → docs=1000 re-verified.
|
||||
|
||||
`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`
|
||||
(full output `/tmp/battery_postfix.log`, pre-fix comparison
|
||||
`/tmp/battery_prefix.log`):
|
||||
|
||||
| # | question | verdict |
|
||||
| --- | --- | --- |
|
||||
| 1 | How do I deploy gitea? | HIGH, `fts_hits=38`; suggested top-5 **byte-identical** to the phase-119 record — `gitea/README.md` **rank 4** ✓ |
|
||||
| 2 | How do I deploy forgejo? | HIGH, `fts_hits=37`; forgejo tree 5/5, `forgejo/README.md` **#1** ✓ |
|
||||
| 3 | How do I set up the gateway? | HIGH, `fts_hits=37`; `gateway/README.md` **rank 3** ✓ |
|
||||
| 4 | Do you have notes on the qwen 3.8 27b model? | HIGH, `fts_hits=31`; both `qwen3.8-27b-*` quadlets at #1/#2 ✓ (the digit-prefix regression holds) |
|
||||
| 5 | What is the capital of Mongolia? | **HIGH, `fts_hits=5`** — unchanged ✓ |
|
||||
| 6 | How do I build a Besta? | **SKIPPED LOUDLY — source absent in the replica snapshot** |
|
||||
| 7 | How do I build the BESTÅ? | **SKIPPED LOUDLY — source absent in the replica snapshot** |
|
||||
|
||||
`SELECT DISTINCT source FROM documents` on the restored replica lists
|
||||
`deploy, homelab, homepage, ServMon, skills, vibe-bot` — NO
|
||||
`ikea_instructions`-class source (`grep -c -i ikea
|
||||
data/bor_live_kb_replica.dump.sql` = 0: the 2026-09-05 snapshot predates
|
||||
the IKEA import, exactly the case D5 anticipated; phase-140's
|
||||
"skipped loudly" precedent). The two questions still ran and did not
|
||||
crash; both return the `build.sh`/`bifrost`-class top-5 that a KB without
|
||||
the product can only return, and neither grounds on a document that names
|
||||
the product. **The rank gate therefore stays with the owner** — it can
|
||||
only close on a KB that has the source (the live KB).
|
||||
|
||||
Behavior-preservation, measured (not argued):
|
||||
|
||||
* the five phase-119 `suggested:` lists are **byte-identical** to the
|
||||
recorded 2026-09-16 run in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md`;
|
||||
* the battery re-run with the task-01 fold REVERTED is **byte-identical**
|
||||
to the post-fix run (`diff` of the question/gate/suggested lines: no
|
||||
difference) — the fold cannot move an ASCII-only corpus, as designed;
|
||||
* `best_cosine` moves by ≤ 0.0004 between the September record and today
|
||||
(real-`embed` model jitter, not code).
|
||||
|
||||
**Verdict: GATE PASS** for questions 1–5 (unchanged), 6–7 recorded as
|
||||
skipped loudly with the reason and the re-check command.
|
||||
|
||||
## 5. Environment left behind
|
||||
|
||||
The replica was restored only for the battery. The pre-restore DB state
|
||||
(backup `/tmp/pre_restore_host.dump`, 13 leftover E2E fixture docs,
|
||||
alembic 0026) is restored afterwards — reload the replica any time with
|
||||
the `restore-test-db` skill.
|
||||
@@ -9,8 +9,9 @@
|
||||
3. **Strictly adhere to the LOCKED DECISIONS** (PLAN §2). If you believe a
|
||||
LOCKED decision is wrong, stop and flag it — do not silently deviate.
|
||||
4. **"One Story, One Phase":** each user story in `.agents/user_stories/`
|
||||
corresponds to a distinct execution phase that includes its own dedicated
|
||||
Playwright E2E test suite (one test file per story, run in isolation).
|
||||
corresponds to a distinct execution phase, and **every** phase — story-driven
|
||||
or owner-directed — includes its own dedicated Playwright E2E test suite
|
||||
(one test file for the phase, run in isolation).
|
||||
5. **"UI Structure Check":** before finalizing any UI component, verify that
|
||||
it follows the layout principles in `.agents/PLAN.md` §7 (proper container
|
||||
usage, the 72rem centered reading column — every page rides the 72rem
|
||||
@@ -57,4 +58,5 @@ uv run pytest --cov=app --cov-report=term-missing
|
||||
uv run playwright install chromium # once
|
||||
uv run pytest tests/e2e/test_smoke.py -v --no-cov
|
||||
uv run ruff check . && uv run pyright # lint + types
|
||||
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt # retrieval regression battery (phases 119/162)
|
||||
```
|
||||
|
||||
@@ -280,7 +280,12 @@ reflect the new files.
|
||||
- A run that changed the KB also regenerates the **KB overview** and the
|
||||
**folder summaries** (the drill-down catalog's stored per-folder
|
||||
descriptions) with the `lite` model — best-effort: a summary failure
|
||||
never fails the import. Because the sync only rewrites the folders a
|
||||
never fails the import. The stored overview ends with a deterministic
|
||||
`Sources: <name> (<k> doc|docs), …` line — every source with its exact
|
||||
document count, catalog order, computed without a model call — so no
|
||||
source (not even a 1-doc one) is invisible to `<knowledge_base>`
|
||||
(phase 162); the `LLMError`/zero-doc paths leave the previous row
|
||||
(tail included) untouched. Because the sync only rewrites the folders a
|
||||
change touches, a one-shot
|
||||
`uv run python -m scripts.regen_folder_summaries` refreshes EVERY
|
||||
stored folder summary in one run (full mode: every candidate folder
|
||||
@@ -492,11 +497,20 @@ server-side tools — on **grounded** (high-relevance) turns only:
|
||||
|------|-------------|
|
||||
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
|
||||
| **`read(path, offset?, limit?)`** | Appends one indexed document's text to context by its combined `source/path`; optional `offset` (1-based first line) + `limit` (line count) read just that part of a large document; very large reads are truncated at `BOR_READ_MAX_CHARS` with a notice that teaches the workflow — grep reports line numbers, then read that part with `offset` + `limit` |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case- and accent-insensitive fixed substring — accents fold: `besta` matches `BESTÅ`, phase 162); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
|
||||
|
||||
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
|
||||
|
||||
**The prompt teaches the search discipline (phase 162).** A summary or an
|
||||
outline is a hint, never an inventory: the system prompt tells the model to run
|
||||
`ls` with no `path` — the full source list — before ever claiming the knowledge
|
||||
base lacks something, and to `grep` the artifact's part number or a filename
|
||||
token. One grep pattern is one string, so a name's split tokens (`frame`,
|
||||
`white`) and its part number (`AA-1272068`) each need their own grep; and a
|
||||
note saying a file is gitignored describes its **repo** status, not whether it
|
||||
is indexed as a source.
|
||||
|
||||
**Git-source provenance (phase 145).** When a document comes from a git
|
||||
source, its `read` result also carries an `origin:` line (the repository's
|
||||
URL, credentials stripped) and, when the source pins one, a `branch:` line.
|
||||
@@ -583,7 +597,12 @@ stemming) and searched **twice** against Postgres:
|
||||
|
||||
1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`)
|
||||
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
|
||||
top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`)
|
||||
top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`), led by
|
||||
**name hits** — documents whose *path* matches a question token (`gitea`
|
||||
→ `gitea/…`, `qwen38` → `qwen3.8-27b-…`; phase 119). Name normalization
|
||||
is **accent-folded** (phase 162): a question's `BESTÅ` folds to `besta`
|
||||
and matches the ASCII filename, and an accented filename (`BÅsta.md`)
|
||||
matches an ASCII question — both directions, stdlib fold only.
|
||||
|
||||
The two ranked lists are fused with **Reciprocal Rank Fusion**
|
||||
(`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores
|
||||
@@ -917,7 +936,7 @@ to `.env` and adjust.
|
||||
| `BOR_MAX_OUTPUT_TOKENS` | `32768` | Max answer length in tokens (answers must not be cut off) |
|
||||
| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `<tuning>` (steering notes) prompt section |
|
||||
| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model |
|
||||
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `<knowledge_base>` prompt section |
|
||||
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `<knowledge_base>` prompt section — the stored outline **plus** its deterministic `Sources: …` tail (phase 162); the cap keeps the longest-fitting prefix, so a value below the default clips the tail first (the default always holds both) |
|
||||
| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation |
|
||||
| `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` | `8000` | Cap on a folder's document list sent to `lite` for folder summaries (phase 94) |
|
||||
|
||||
@@ -1058,8 +1077,9 @@ normally until you attach. Override the port with `DEBUGPY_PORT`.
|
||||
|
||||
## QA / Testing
|
||||
|
||||
Three layers — the project rule is **one story, one phase, one Playwright
|
||||
suite** (see `AGENTS.md`):
|
||||
Three layers — the project rule is **one phase, one Playwright suite** (a
|
||||
phase may implement a user story or an owner-directed change; see
|
||||
`AGENTS.md`):
|
||||
|
||||
```bash
|
||||
# Unit + integration (FastAPI TestClient)
|
||||
@@ -1075,12 +1095,27 @@ uv run pyright
|
||||
# Playwright E2E — install the browser once:
|
||||
uv run playwright install chromium
|
||||
|
||||
# Each story's E2E runs IN ISOLATION:
|
||||
# Each phase's E2E runs IN ISOLATION:
|
||||
uv run pytest tests/e2e/test_import_documents.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
|
||||
# ...one file per story in .agents/user_stories/ (see .agents/phases/todo/)
|
||||
# ...one file per phase in tests/e2e/ (the stories live in
|
||||
# .agents/user_stories/, see .agents/phases/todo/)
|
||||
```
|
||||
|
||||
**Retrieval regression battery.** `tests/fixtures/retrieval_battery.txt` is
|
||||
the tracked retrieval gate (phases 119 + 162): the name-your-tool questions,
|
||||
the versioned-name (`qwen 3.8 27b`) regression, the out-of-KB deflection
|
||||
sanity, and the `besta` / `BESTÅ` accent-fold pair. Run it against a KB with
|
||||
`scripts.eval_retrieval` (the real pipeline, real embeddings):
|
||||
|
||||
```bash
|
||||
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
|
||||
```
|
||||
|
||||
The exact expected ranks are recorded in the fixture's header comments; a
|
||||
question whose source is missing from the KB snapshot is recorded as skipped
|
||||
loudly, never silently dropped.
|
||||
|
||||
**Deterministic E2E:** by default the E2E app talks to a local **mock
|
||||
aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real token-overlap
|
||||
vectors — so the cosine relevance threshold behaves like production. To
|
||||
|
||||
+21
-11
@@ -131,8 +131,10 @@ task 04):
|
||||
the only capped read (the phase-95 cap unchanged). And ``grep``
|
||||
greps the
|
||||
indexed documents (or the one document a combined ``source/path``
|
||||
names) for a case-insensitive fixed substring and returns up to 20
|
||||
``source/path:line: text`` match lines (owner-locked A5, phase 68),
|
||||
names) for a case- and accent-insensitive fixed substring (accents
|
||||
fold — ``besta`` matches ``BESTÅ``; phase 162, LOCKED A3) and
|
||||
returns up to 20 ``source/path:line: text`` match lines
|
||||
(owner-locked A5, phase 68),
|
||||
each line truncated to 200 chars. A grep is a **locator**, not a
|
||||
context-adder: it never appends to the answer context (only
|
||||
``read`` does — ``holder.read_docs`` is untouched by a grep).
|
||||
@@ -312,7 +314,7 @@ from app.rag.llm import (
|
||||
ToolResultPiece,
|
||||
chat_stream_retried,
|
||||
)
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from app.rag.retriever import TRUNCATION_MARKER, _fold
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
from app.rag.source_removal import resolve_source_name
|
||||
|
||||
@@ -437,7 +439,8 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
"name": "grep",
|
||||
"description": (
|
||||
"Search the indexed documents for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines "
|
||||
"(case- and accent-insensitive — accents fold: `besta` "
|
||||
"matches `BESTÅ`) and return up to 20 matching lines "
|
||||
"as `source/path:line: text` — a locator, not a "
|
||||
"context-adder: read the winner with `read` — for a "
|
||||
"large document, use the reported line number as "
|
||||
@@ -458,7 +461,9 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
"type": "string",
|
||||
"description": (
|
||||
"The exact text to search for (a plain "
|
||||
"substring, not a regex — no '.*', no "
|
||||
"substring — case- and accent-insensitive, "
|
||||
"accents fold: `besta` matches `BESTÅ` — "
|
||||
"not a regex — no '.*', no "
|
||||
"'\\.', no character classes)"
|
||||
),
|
||||
},
|
||||
@@ -1560,16 +1565,21 @@ def format_covered_spans(spans: Sequence[tuple[int, int]]) -> str:
|
||||
def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
|
||||
"""Every line of *content* that contains *pattern*, in file order.
|
||||
|
||||
Case-insensitive **fixed substring** (owner-locked A5: no regex — no
|
||||
ReDoS surface, a simple contract for the model). Returns
|
||||
``(1-based line number, line.rstrip())`` pairs; an empty *content*
|
||||
never matches a non-empty pattern.
|
||||
Case- **and accent-insensitive fixed substring** (owner-locked A5:
|
||||
no regex — no ReDoS surface, a simple contract for the model). Both
|
||||
sides fold through the shared :func:`app.rag.retriever._fold` plus
|
||||
lowercase (phase 162, LOCKED A3 — the fold changes the alphabet it
|
||||
ignores, not the matching semantics: ``grep "besta"`` matches a
|
||||
content line ``The BESTÅ frame``, and a pattern carrying accents
|
||||
folds too), so grep and the name-hit path can never drift apart.
|
||||
Returns ``(1-based line number, line.rstrip())`` pairs; an empty
|
||||
*content* never matches a non-empty pattern.
|
||||
"""
|
||||
needle = pattern.lower()
|
||||
needle = _fold(pattern.lower())
|
||||
return [
|
||||
(number, line.rstrip())
|
||||
for number, line in enumerate(content.split("\n"), start=1)
|
||||
if needle in line.lower()
|
||||
if needle in _fold(line.lower())
|
||||
]
|
||||
|
||||
|
||||
|
||||
+63
-7
@@ -28,6 +28,21 @@ an import added/updated at least one document (or no row exists yet),
|
||||
and a ``lite`` failure logs and leaves the previous outline intact — an
|
||||
old outline is better than none (phase locked decisions).
|
||||
|
||||
After the model's outline, :func:`regenerate_overview` appends ONE
|
||||
deterministic, non-LLM line — ``Sources: <name> (<k> doc|docs), …``
|
||||
(phase 162, LOCKED A2). The 2026-10-07 live finding: the ``lite``
|
||||
outline is lossy — it listed 3 groups of a 14-source index, so the
|
||||
1-doc sources (``ikea_instructions``, ``players_handbook``) were
|
||||
invisible to ``<knowledge_base>``. Small sources must be visible
|
||||
without a model call, so the line is computed mechanically from the
|
||||
catalogue rows (:func:`_source_inventory`) — the mock/real model
|
||||
outputs never affect it, and ``build_overview_prompt`` never sees the
|
||||
tail (the model input stays byte-identical). The longest-prefix cap in
|
||||
``build_kb_section`` applies to the WHOLE stored text: at the default
|
||||
``BOR_KB_OVERVIEW_MAX_CHARS`` (4000) the ~1500-char outline plus the
|
||||
one-line tail (~15 chars/source) always fit; a smaller owner-set cap
|
||||
cuts the tail first — accepted, the cap is owner-tunable.
|
||||
|
||||
The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` /
|
||||
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it
|
||||
in the system prompt (wired in phase 31, task 05).
|
||||
@@ -101,6 +116,35 @@ def _first_summary_line(summary: str | None) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str:
|
||||
"""The phase-162 deterministic source inventory (LOCKED A2).
|
||||
|
||||
Mechanical, non-LLM: one entry per **distinct** source in catalog
|
||||
order (``source, path`` — first occurrence in the ``source,
|
||||
path``-ordered rows), each ``<name> (<k> doc|docs)`` — singular
|
||||
``doc`` at k=1, ``docs`` otherwise — joined with ``, `` behind the
|
||||
``Sources: `` prefix. The ``lite`` outline stays the model's job;
|
||||
this line guarantees no source — not even a 1-doc one like the
|
||||
2026-10-07 ``ikea_instructions`` case — can be invisible to
|
||||
``<knowledge_base>``, whatever the model outputs. The mock/real
|
||||
model outputs never affect it.
|
||||
|
||||
*rows* is the existing ``(source, path, title, summary)``
|
||||
projection :func:`regenerate_overview` already reads — no new
|
||||
query. At the default ``BOR_KB_OVERVIEW_MAX_CHARS`` (4000) the
|
||||
outline plus this one line always fit; a smaller owner-set cap
|
||||
cuts the tail first (accepted — the cap is owner-tunable).
|
||||
"""
|
||||
counts: dict[str, int] = {}
|
||||
for source, _path, _title, _summary in rows:
|
||||
counts[source] = counts.get(source, 0) + 1
|
||||
entries = ", ".join(
|
||||
f"{name} ({count} {'doc' if count == 1 else 'docs'})"
|
||||
for name, count in sorted(counts.items())
|
||||
)
|
||||
return f"Sources: {entries}"
|
||||
|
||||
|
||||
def build_overview_prompt(
|
||||
rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None
|
||||
) -> tuple[str, str]:
|
||||
@@ -166,10 +210,16 @@ async def regenerate_overview(
|
||||
|
||||
* Zero documents → the existing row (if any) is left untouched and
|
||||
``False`` is returned — no KB, no outline, no wasted model call.
|
||||
* Happy path → the model's text is upserted into the single row
|
||||
(``id = 1``, fresh UTC ``updated_at``), committed, and
|
||||
``overview: regenerated docs=… chars=…`` is logged; returns
|
||||
``True``. Idempotent by construction: one row, always ``id = 1``.
|
||||
* Happy path → the model's text (trailing newlines rstripped) plus
|
||||
the deterministic ``Sources: …`` inventory line (phase 162,
|
||||
LOCKED A2 — :func:`_source_inventory`) is upserted into the
|
||||
single row (``id = 1``, fresh UTC ``updated_at``), committed, and
|
||||
``overview: regenerated docs=… chars=…`` is logged (chars = the
|
||||
FULL stored content, tail included); returns ``True``. Idempotent
|
||||
by construction: one row, always ``id = 1``. The model input
|
||||
never sees the tail; the zero-doc and ``LLMError`` paths below
|
||||
leave a previous row byte-identical — an existing tail is part of
|
||||
the stored content and is preserved on failure.
|
||||
* :class:`LLMError` → ``overview: regeneration failed — …`` is
|
||||
logged and ``False`` is returned with the previous row intact —
|
||||
an old outline is better than none.
|
||||
@@ -209,15 +259,21 @@ async def regenerate_overview(
|
||||
logger.error("overview: regeneration failed — %s", e)
|
||||
return False
|
||||
|
||||
# Phase 162 (LOCKED A2): the deterministic source inventory is
|
||||
# appended to the STORED content after the model's outline —
|
||||
# mechanical, non-LLM, so no source can be invisible to
|
||||
# `<knowledge_base>`. The model input above never sees the tail.
|
||||
content = text.rstrip("\n") + "\n" + _source_inventory(rows)
|
||||
|
||||
now = datetime.now(UTC)
|
||||
row = session.get(KbOverview, 1)
|
||||
if row is None:
|
||||
session.add(KbOverview(id=1, content=text, updated_at=now))
|
||||
session.add(KbOverview(id=1, content=content, updated_at=now))
|
||||
else:
|
||||
row.content = text
|
||||
row.content = content
|
||||
row.updated_at = now
|
||||
session.commit()
|
||||
logger.info("overview: regenerated docs=%d chars=%d", len(rows), len(text))
|
||||
logger.info("overview: regenerated docs=%d chars=%d", len(rows), len(content))
|
||||
return True
|
||||
finally:
|
||||
if owns_session:
|
||||
|
||||
+30
-2
@@ -265,6 +265,31 @@ ORIGIN_RULE: str = (
|
||||
"again for the source."
|
||||
)
|
||||
|
||||
#: The search-discipline rule (phase 162, task 03, LOCKED A4) — appended
|
||||
#: to the ``TOOLS_SECTION`` tools paragraph immediately after the grep
|
||||
#: sentence (i.e. after the ``…retry with the plain text you expect to
|
||||
#: see.`` clause, the placement from the 2026-10-07 owner conversation
|
||||
#: where the agent answered from the lossy ``<knowledge_base>`` outline
|
||||
#: instead of running ``ls``, and where the exact-substring grep silently
|
||||
#: missed the accented ``BESTÅ`` in the IKEA PDF's text layer). Adapted
|
||||
#: verbatim from the conversation's proposed system-prompt patch for the
|
||||
#: folded grep (LOCKED A3). The three lessons: (1) an outline/summary is
|
||||
#: a hint, never an inventory — ``ls`` (no ``path``) before claiming the
|
||||
#: KB lacks something; (2) grep folds accents but one pattern is one
|
||||
#: string — split filename tokens and the part number need separate
|
||||
#: greps; (3) a gitignored note describes repo status, not index status.
|
||||
#: The mock E2E is unaffected (it keys on markers, not this sentence).
|
||||
SEARCH_DISCIPLINE_RULE: str = (
|
||||
"A summary or outline is a hint, never an inventory: before claiming "
|
||||
"the knowledge base lacks something, run `ls` with no `path` to list "
|
||||
"every source, and `grep` the artifact's part number or a filename "
|
||||
"token. `grep` folds accents — `besta` matches `BESTÅ` — but one "
|
||||
"pattern is one string: a name's split tokens (`frame`, `white`) and "
|
||||
"its part number (`AA-1272068`) need separate greps. A note saying a "
|
||||
"file is gitignored describes its repo status, not whether it is "
|
||||
"indexed as a source."
|
||||
)
|
||||
|
||||
TOOLS_SECTION: str = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
@@ -305,12 +330,15 @@ TOOLS_SECTION: str = (
|
||||
"path as plain text otherwise. The user should be able to open the "
|
||||
"complete original from your answer in one step, not ask again for "
|
||||
"the source. `grep` locates an exact string "
|
||||
"(case-insensitive) in the indexed documents and returns up to 20 "
|
||||
"(case- and accent-insensitive — accents fold: `besta` matches "
|
||||
"`BESTÅ`) in the indexed documents and returns up to 20 "
|
||||
"matching `source/path:line: text` lines — a locator, not a "
|
||||
"context-adder: read the winner with `read`. A grep pattern is a "
|
||||
"plain substring, NEVER a regex — '.*' and '\\.' are literal text "
|
||||
"there; if such a pattern returns no matches, retry with the plain "
|
||||
"text you expect to see. For a normal search pass only `pattern` — "
|
||||
"text you expect to see. "
|
||||
+ SEARCH_DISCIPLINE_RULE
|
||||
+ " For a normal search pass only `pattern` — "
|
||||
"its optional `path` argument "
|
||||
"limits the search to one document you already know, by the same "
|
||||
"combined `source/path` string; never a source name — a bare "
|
||||
|
||||
+63
-6
@@ -28,9 +28,21 @@
|
||||
order ``(source, path)`` — the old total-matched-length tie-break
|
||||
is retired, it outranked 5-char product names by 6-char common
|
||||
tokens; capped at :data:`NAME_HIT_LIMIT`),
|
||||
the FTS rows follow. This is what finds name-your-tool questions
|
||||
("gitlab") that vector similarity buries — and, the 2026-09-05
|
||||
incident, the versioned-name case the default parser lexes
|
||||
the FTS rows follow. NAME NORMALIZATION IS ACCENT-FOLDED (phase 162,
|
||||
LOCKED A1): :func:`_normalize_name` runs :func:`_fold` (NFKD
|
||||
decomposition, combining marks dropped — stdlib only) before the
|
||||
alnum scan, and it is the SINGLE normalization both sides of name
|
||||
matching pass through (question tokens and path parts alike) — the
|
||||
2026-10-07 live gap this closes: the IKEA filename is ASCII
|
||||
(``besta-frame-white__AA-1272068-11-100``) while its content layer
|
||||
spells the name only as ``BESTÅ`` (U+00C5), and the old lower-only
|
||||
normalization turned the accented question token into ``best`` —
|
||||
never a match. With the fold, ``BESTÅ`` → ``besta`` (was ``best``)
|
||||
and an accented filename (``BÅsta.md`` → ``basta``) answers to the
|
||||
ASCII question — one function, both directions. This is what finds
|
||||
name-your-tool questions ("gitlab") that vector similarity buries —
|
||||
and, the 2026-09-05 incident, the versioned-name case the default
|
||||
parser lexes
|
||||
incompatibly (question "Qwen 3.8" → ``qwen``/``3``/``8`` can never
|
||||
match a document's ``qwen3``/``8``/``27b`` tokens, while every
|
||||
unrelated llama.cpp quadlet out-ranks the target on the shared
|
||||
@@ -81,6 +93,7 @@ from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
import unicodedata
|
||||
import uuid
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, replace
|
||||
@@ -130,9 +143,42 @@ _ALNUM_RE = re.compile(r"[a-z0-9]+")
|
||||
_STEM_SPLIT_RE = re.compile(r"[^a-z0-9]+")
|
||||
|
||||
|
||||
def _fold(s: str) -> str:
|
||||
"""The shared accent fold (phase 162, LOCKED A1): NFKD decomposition,
|
||||
combining marks dropped (stdlib ``unicodedata`` + ``str.casefold``
|
||||
only — no new dependency).
|
||||
|
||||
``Å/å → a``, ``Ä/Ö/Ü → a/o/u``, ``ß → ss``, ``fi → fi``, ``ç → c``
|
||||
(``casefold`` is the Unicode fold — the ``ß → ss`` case rides it:
|
||||
NFKD alone leaves ``ß`` intact — and ``str.lower`` leaves the ``fi``
|
||||
ligature). Characters with no decomposition (CJK) survive the fold
|
||||
and still drop out under the caller's alnum scan (behavior
|
||||
unchanged for them); compatibility-decomposable characters (``①`` →
|
||||
``1``) fold to their ASCII base, which :data:`NAME_TOKEN_MIN_LEN`
|
||||
already excludes as a name token (a digit base can still feed the
|
||||
versioned join as the numeric NEXT token — the only route left is a
|
||||
prefix-class candidate no real path matches). Case is folded too
|
||||
(the caller lowercases anyway). Imported by ``app.rag.agent`` for
|
||||
the folded ``grep`` (phase 162, LOCKED A3) — one fold, never
|
||||
drifting apart.
|
||||
"""
|
||||
decomposed = unicodedata.normalize("NFKD", s.casefold())
|
||||
return "".join(c for c in decomposed if not unicodedata.combining(c))
|
||||
|
||||
|
||||
def _normalize_name(s: str) -> str:
|
||||
"""Lowercased, alnum-only form of *s* (``Qwen 3.8`` → ``qwen38``)."""
|
||||
return "".join(_ALNUM_RE.findall(s.lower()))
|
||||
"""Lowercased, ACCENT-FOLDED, alnum-only form of *s* (``Qwen 3.8`` →
|
||||
``qwen38`` unchanged; ``BESTÅ`` → ``besta`` — the phase-162 fold).
|
||||
|
||||
This is the SINGLE normalization both sides of name matching pass
|
||||
through — the :func:`name_hit_tokens` question tokens and the
|
||||
:func:`_name_parts` path parts / file stem / stem sub-components —
|
||||
so the one change bridges both directions (the ASCII question
|
||||
``besta`` keeps matching the ASCII filename, the accented question
|
||||
``BESTÅ`` now matches it too, and an accented filename ``BÅsta.md``
|
||||
answers to the ASCII question ``basta``).
|
||||
"""
|
||||
return "".join(_ALNUM_RE.findall(_fold(s.lower())))
|
||||
|
||||
|
||||
def name_hit_tokens(question: str) -> list[str]:
|
||||
@@ -166,7 +212,10 @@ def name_hit_tokens(question: str) -> list[str]:
|
||||
Order of first appearance, de-duplicated. Matching applies the
|
||||
A2 rule: digit-bearing candidates prefix a normalized path part
|
||||
or file stem; digitless candidates equal a part, stem, or stem
|
||||
sub-component — titles are never matched.
|
||||
sub-component — titles are never matched. Candidates are
|
||||
ACCENT-FOLDED through :func:`_normalize_name` (phase 162) — the
|
||||
token ``besta`` drawn from ``BESTÅ`` name-matches ``BESTÅ``-class
|
||||
names both ways.
|
||||
"""
|
||||
tokens = question.split()
|
||||
out: list[str] = []
|
||||
@@ -456,6 +505,10 @@ def _name_parts(path: str) -> tuple[set[str], set[str]]:
|
||||
(the stem lowercased, split on non-alphanumeric runs, each piece
|
||||
normalized, empties dropped: ``kubernetes_gitea`` →
|
||||
``kubernetes`` / ``gitea``): DIGITLESS tokens exact-match these.
|
||||
|
||||
The parts are ACCENT-FOLDED through :func:`_normalize_name` (phase
|
||||
162) — ``besta`` name-matches ``BESTÅ``-class names both ways (the
|
||||
accented stem ``BÅsta`` enters the equal set as ``basta``).
|
||||
"""
|
||||
p = Path(path)
|
||||
prefix = {_normalize_name(part) for part in p.parts}
|
||||
@@ -479,6 +532,10 @@ def _name_token_matches(token: str, prefix: set[str], equal: set[str]) -> bool:
|
||||
* the token HAS NO DIGIT → it EQUALS a normalized path part, file
|
||||
stem, or stem sub-component (``gitea`` → the ``gitea/`` folder,
|
||||
``gitea.md``, ``kubernetes_gitea``, ``gitea-values``).
|
||||
|
||||
Both the token and the path sets arrive ACCENT-FOLDED from
|
||||
:func:`_normalize_name` (phase 162) — ``besta`` matches
|
||||
``BESTÅ``-class names both ways.
|
||||
"""
|
||||
if any(ch.isdigit() for ch in token):
|
||||
return any(part.startswith(token) for part in prefix)
|
||||
|
||||
@@ -307,8 +307,11 @@ def test_mock_generated_outline_is_stored_and_echoed(
|
||||
assert user.splitlines()[0].startswith(
|
||||
"docs — homelab/tables.md — Service Port Table"
|
||||
), "the generator's list must lead with the NEWEST doc (recency-first)"
|
||||
# Phase 162 (LOCKED A2): the stored content ends with the
|
||||
# deterministic source-inventory line — one source, 13 docs.
|
||||
expected = (
|
||||
"Knowledge base outline:\n- docs homelab tables md service port table this"
|
||||
"Knowledge base outline:\n- docs homelab tables md service port table this\n"
|
||||
"Sources: docs (13 docs)"
|
||||
)
|
||||
assert row.content == expected
|
||||
first_bullet = row.content.splitlines()[1].removeprefix("- ").strip()
|
||||
|
||||
@@ -329,8 +329,10 @@ def test_regenerated_outline_leads_with_newest_doc(
|
||||
assert row is not None and row.id == 1
|
||||
# The stored outline is the mock's 8-token digest of the capped list
|
||||
# — all 8 tokens from the NEWEST doc's source/path/title (computed
|
||||
# from the fixture above, not hardcoded wording):
|
||||
assert row.content == EXPECTED_DIGEST
|
||||
# from the fixture above, not hardcoded wording) — and phase 162
|
||||
# (LOCKED A2) appends the deterministic source-inventory line after
|
||||
# it: every fixture source named with its exact count.
|
||||
assert row.content == EXPECTED_DIGEST + "\nSources: reese (5 docs)"
|
||||
assert "old" not in row.content # no oldest-batch token survives the cut
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,864 @@
|
||||
"""Phase 162 E2E (Playwright, mock-only): a 1-document source is BOTH
|
||||
visible to ``<knowledge_base>`` and winnable by its accented question.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov
|
||||
|
||||
The 2026-10-07 live miss, replayed deterministically: the owner asked
|
||||
``How do I build the BESTÅ?``; the KB held ``ikea_instructions/
|
||||
besta-frame-white__AA-1272068-11-100.md`` (1 document in the source, its
|
||||
body carrying the name ONLY as ``BESTÅ``) and the question came back
|
||||
empty-handed — ASCII ``besta`` documents outranked the one document that
|
||||
actually named the product. Three separate losses, three pins here:
|
||||
|
||||
* **D1 (the question-side fold)** — name matching never bridged the
|
||||
accent: ``BESTÅ`` normalized to ``best``, ``besta-frame-…`` to
|
||||
``besta``, and the digitless rule EQUALS a stem component. Test 1
|
||||
pins the fix, test 2 MEASURES the miss.
|
||||
* **D2 (the deterministic source inventory)** — the ``lite`` outline is
|
||||
lossy, so a 1-doc source could be invisible to the model. Test 4 pins
|
||||
the mechanical ``Sources:`` tail on the stored row.
|
||||
* **D3 (the grep fold + search discipline)** — the stored accented body
|
||||
stayed grep-matchable by the ASCII name. Test 6 pins the content
|
||||
layer; the turn-level prose discipline is the prompt teaching
|
||||
(``app/rag/prompts.py``, unit-pinned in task 03).
|
||||
|
||||
KB fixture — ``tests/fixtures/recallkb`` (tracked), THREE local-directory
|
||||
sources registered through the authenticated API + the REAL in-process
|
||||
``POST /api/sync`` (the ``test_name_signal_read_chips.py`` pattern), six
|
||||
markdown documents whose text is token-controlled so the hybrid gate, the
|
||||
FTS list and the name-hit bonus pick the intended tiers deterministically
|
||||
(``synced_kb`` pins the tiers with the app's REAL retrieval — a
|
||||
fixture-text regression breaks at setup with a measured message):
|
||||
|
||||
* ``ikea_instructions/`` — the small source: ONE document, the live
|
||||
incident's own filename, its body naming the product ONLY as the
|
||||
accented ``BESTÅ`` (no ASCII ``besta`` anywhere in the file). Two
|
||||
token controls make the mechanism visible: no ``build`` (the question's
|
||||
other FTS token) anywhere, and the accented name deliberately placed
|
||||
PAST the 24-token digest window, so the mock summary chunk leaks no
|
||||
ASCII ``best`` lexeme that a pre-fix FTS could have hit;
|
||||
* ``phase_docs/`` (4 docs) + ``players_handbook/`` (1 doc) — the ASCII
|
||||
distractors of the live index: every body carries ASCII ``besta`` AND
|
||||
``build``, so each of the five outranks the small source on
|
||||
pre-fix retrieval (two fused terms — FTS + vector — against its one).
|
||||
|
||||
Measured, never assumed (test 2 re-measures both directions; the same
|
||||
numbers are recorded in
|
||||
``.agents/reports/162_search_recall_small_sources/``). Accented question
|
||||
``How do I build the BESTÅ?``, the app's real ``retrieve()`` +
|
||||
``select_suggested()`` over this KB with the mock's embeddings (gate:
|
||||
threshold 0.30, lexical-support floor 0.15 — the conftest mock
|
||||
calibration; ``best cosine`` / ``fts`` are the A8 inputs):
|
||||
|
||||
PRE-FIX (the phase-119 ``_normalize_name``, no accent fold)
|
||||
name tokens ['build', 'best'] → NO name hit
|
||||
suggested the five ASCII distractors
|
||||
related ikea_instructions/besta-frame-white__… (rank 6)
|
||||
best cosine 0.382, fts 8 (grounded, on the WRONG documents —
|
||||
the live 2026-10-07 shape: an answer
|
||||
about the ASCII ``besta`` notes)
|
||||
POST-FIX (shipped)
|
||||
name tokens ['build', 'besta'] → name hit
|
||||
suggested THE SMALL SOURCE #1, then four distractors
|
||||
related phase_docs/156_history_source_filter.md
|
||||
best cosine 0.382, fts 9 (the fold adds the lexical term; the
|
||||
cosine is untouched — LOCKED A1)
|
||||
|
||||
The mechanism is the term COUNT: every distractor matches the question's
|
||||
other FTS token ``build``, so each fuses TWO terms (vector + FTS) while
|
||||
the small source — no ``build``, no accent-bridgeable lexeme — fuses ONE
|
||||
and lands rank 6. The folded name hit is the only signal that can lift
|
||||
it: name hits LEAD the lexical list (``fuse`` rank 1 ⇒ 1/61) and the
|
||||
per-document ``BOR_NAME_HIT_BONUS`` (0.005) rides on top. The ASCII
|
||||
question ``How do I build a Besta?`` keeps the same document suggested
|
||||
#1 with the same cosine, before and after (the untouched phase-119
|
||||
path, test 3).
|
||||
|
||||
Test → contract mapping (task 04 cases 1–4):
|
||||
1. ``test_accented_question_suggests_the_small_source`` — (1): the
|
||||
accented question grounds (``deflected: false``), the durable
|
||||
record's SUGGESTED slice (entries 1–5, LOCKED A3) contains
|
||||
``ikea_instructions/besta-frame-white__AA-1272068-11-100.md``, and
|
||||
the ``done`` frame's ``related`` list — the rank-6+ remainder —
|
||||
does NOT (the doc moved from the rank-6 miss into the tier). A
|
||||
zero-read grounded turn chips nothing (phase 119 LOCKED A1).
|
||||
2. ``test_under_rank_is_measured_not_assumed`` — (1, the hypothesis):
|
||||
with the PRE-FIX normalizer re-installed (``monkeypatch``), the same
|
||||
retrieval over the same KB puts the doc OUT of the top-5 (and the
|
||||
``BESTÅ?`` token degrades to ``best``); with the shipped normalizer
|
||||
it is suggested #1. The under-rank is measured in both directions,
|
||||
so the pin can never silently rot.
|
||||
3. ``test_ascii_question_still_name_matches`` — (2): the ASCII
|
||||
regression (the phase-119 path untouched) — same document in the
|
||||
suggested tier, grounded, same related remainder.
|
||||
4. ``test_overview_tail_names_every_source_with_counts`` — (3): the
|
||||
stored ``kb_overview`` row ENDS with the deterministic
|
||||
``Sources: …`` line, one entry per source with exact counts — the
|
||||
1-doc sources visible whatever the ``lite`` digest said.
|
||||
5. ``test_out_of_kb_question_still_deflects`` — (4): the gate verdict
|
||||
for the out-of-KB question is UNCHANGED (``deflected: true``,
|
||||
``fts_hits == 0``, no chips, ``Maybe try`` chips only) — the fold
|
||||
must not turn a real miss into a false grounding.
|
||||
6. ``test_grep_folds_the_accented_body`` — (3, content layer):
|
||||
``grep_document`` with the ASCII pattern ``besta`` matches the
|
||||
stored accented line (D3), and the accented pattern matches too.
|
||||
7. ``test_suggested_folder_lines_carry_the_small_source`` — the
|
||||
prompt-level witness: the mock's ``repeat your folder map`` echo
|
||||
ends with the suggested-folder lines, and the 1-doc source's line
|
||||
``ikea_instructions/:`` is among them (its suggested doc owns the
|
||||
line, so the header stands alone).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document, KbOverview, QueryLog
|
||||
from app.rag import retriever
|
||||
from app.rag.agent import grep_document
|
||||
from app.rag.retriever import retrieve, select_related, select_suggested
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import TOKEN_RE, embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: Own port (the phase-79 full-inventory rule: a module app binds its
|
||||
#: own port so a combined run never drives the session app).
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_RECALL", "8162"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "recallkb"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The fixture KB (tracked files on disk — read back byte-for-byte)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: source name → document paths. THE SMALL SOURCE is the live incident:
|
||||
#: one document whose body carries the product name only accented.
|
||||
SOURCE_DOCS: dict[str, list[str]] = {
|
||||
"ikea_instructions": ["besta-frame-white__AA-1272068-11-100.md"],
|
||||
"phase_docs": [
|
||||
"140_pictorial_pdf_pages.md",
|
||||
"148_pdf_text_backfill.md",
|
||||
"152_seaweedfs_blob_ingest.md",
|
||||
"156_history_source_filter.md",
|
||||
],
|
||||
"players_handbook": ["game_night_setup.md"],
|
||||
}
|
||||
|
||||
IKEA_SOURCE = "ikea_instructions"
|
||||
IKEA_DOC = "besta-frame-white__AA-1272068-11-100.md"
|
||||
IKEA = f"{IKEA_SOURCE}/{IKEA_DOC}"
|
||||
|
||||
#: Every fixture doc in ``(source, path)`` catalog order.
|
||||
ALL_DOCS = [f"{source}/{path}" for source in SOURCE_DOCS for path in SOURCE_DOCS[source]]
|
||||
|
||||
#: The deterministic source inventory tail (LOCKED A2 order: name-sorted,
|
||||
#: singular ``doc`` at k=1) — the D2 assertion.
|
||||
INVENTORY = "Sources: ikea_instructions (1 doc), phase_docs (4 docs), players_handbook (1 doc)"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The scripted questions
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The live miss, byte-for-byte (the ``Å`` is the whole point — the file
|
||||
#: spells it ``BESTÅ``, the pre-phase name matcher read ``best``).
|
||||
ACCENTED_QUESTION = "How do I build the BESTÅ?"
|
||||
#: The phase-119 regression: the same product, spelled in ASCII. This
|
||||
#: question ALWAYS name-matched (the untouched path).
|
||||
ASCII_QUESTION = "How do I build a Besta?"
|
||||
#: The accented question + the mock's folder-map trigger — the prompt
|
||||
#: lens on the suggested tier (the folder lines are composed from the
|
||||
#: suggested documents only).
|
||||
FOLDER_QUESTION = ACCENTED_QUESTION + " repeat your folder map"
|
||||
#: The out-of-KB control (the phase-119 battery's deflection sanity).
|
||||
DEFLECT_QUESTION = "What is the capital of Mongolia?"
|
||||
|
||||
|
||||
def _expected_summary(content: str, source: str, path: str) -> str:
|
||||
"""The mock lite model's byte-stable digest + the code pointer line
|
||||
(mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch — the
|
||||
first 24 tokens of the document content — plus the summarizer's
|
||||
deterministic ``Source:`` line; no model output is ever trusted)."""
|
||||
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
|
||||
return f"This document covers {digest}.\nSource: {source}/{path}"
|
||||
|
||||
|
||||
def _prefix_normalizer(value: str) -> str:
|
||||
"""The PRE-PHASE-162 name normalizer (the measured before-state):
|
||||
lower + alnum scan with NO accent fold — the exact body
|
||||
``_normalize_name`` had before task 01. Installed by monkeypatch in
|
||||
test 2 so the miss is MEASURED, not asserted from reasoning."""
|
||||
return "".join(retriever._ALNUM_RE.findall(value.lower()))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seed_dirs() -> list[Path]:
|
||||
"""The three local-directory sources (the registered directory's
|
||||
BASENAME is the source name, phase 38). The small source must be a
|
||||
directory of its own — that is what makes it a 1-doc source."""
|
||||
dirs = []
|
||||
for source, paths in SOURCE_DOCS.items():
|
||||
directory = FIXTURES / source
|
||||
assert directory.is_dir(), f"{directory} is missing"
|
||||
for path in paths:
|
||||
assert (directory / path).is_file(), f"fixture doc {source}/{path} is missing"
|
||||
dirs.append(directory)
|
||||
return dirs
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, seed_dirs: list[Path]) -> Iterator[str]:
|
||||
"""The real app under test — per-module app on its own port (the
|
||||
phase-79 inventory pattern), the mock LLM, the mock-calibrated gate,
|
||||
and the code defaults for everything the tiers depend on (suggested
|
||||
5 / related 2 / name-hit bonus 0.005 / recency 0.0007, exactly the
|
||||
production ``.env``-free defaults)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (the conftest pattern): the accented and the
|
||||
# ASCII turns ground on the distractors' cosine (≈0.35); the
|
||||
# out-of-KB control stays under it.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env is the owner's — empty the git list so the registry
|
||||
# holds EXACTLY the three directories this suite registers, and pin
|
||||
# the leak-guarded corpus settings (the conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the tier margins
|
||||
are pinned against EXACTLY these six documents."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the cookie-authenticated status endpoint to a terminal state
|
||||
(the ``test_summary_seed_context`` pattern over plain httpx)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _tiers(question: str) -> tuple[list[str], list[str], float, int]:
|
||||
"""The measured tiers of one question over the synced KB, with the
|
||||
app's REAL hybrid retrieval and the mock's embeddings (the
|
||||
phase-119 measurement lens): ``(suggested, related, top cosine,
|
||||
fts hits)``."""
|
||||
with SessionLocal() as db:
|
||||
chunks = retrieve(db, question, embed_text(question))
|
||||
suggested = select_suggested(chunks)
|
||||
related = select_related(
|
||||
chunks,
|
||||
{d.id for d in suggested},
|
||||
_Settings.model_fields["related_max_docs"].default,
|
||||
)
|
||||
return (
|
||||
[f"{d.source}/{d.path}" for d in suggested],
|
||||
[f"{d.source}/{d.path}" for d in related],
|
||||
max((c.cosine for c in chunks), default=0.0),
|
||||
sum(1 for c in chunks if c.fts_hit),
|
||||
)
|
||||
|
||||
|
||||
def _assert_tiers(
|
||||
question: str, suggested: list[str], related: list[str]
|
||||
) -> None:
|
||||
"""Pin the engineered invariant for one scripted question, with a
|
||||
MEASURED message (a fixture-text regression fails at setup, not at
|
||||
the wire assertions)."""
|
||||
got_suggested, got_related, cosine, fts = _tiers(question)
|
||||
assert got_suggested == suggested, (
|
||||
f"suggested tier drifted for {question!r}: {got_suggested} "
|
||||
f"(expected {suggested}; best cosine {cosine:.3f}, fts {fts})"
|
||||
)
|
||||
assert got_related == related, (
|
||||
f"related tier drifted for {question!r}: {got_related} "
|
||||
f"(expected {related})"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, seed_dirs: list[Path]) -> None:
|
||||
"""The story's precondition: the three-source KB synced under the
|
||||
deterministic mock. Registers the directories through the
|
||||
authenticated API, runs the REAL in-process sync, pins the stored
|
||||
content byte-identical to the tracked fixture files (including the
|
||||
accented ``BESTÅ`` surviving the round trip untouched), pins the
|
||||
fixture's own token controls (the mechanism the miss is reproduced
|
||||
with), and pins the engineered tiers for all three scripted
|
||||
questions."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
for directory in seed_dirs:
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(directory)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == len(ALL_DOCS), body["detail"]
|
||||
assert body["detail"]["pruned"] == 0, body["detail"]
|
||||
|
||||
with SessionLocal() as db:
|
||||
for source, paths in SOURCE_DOCS.items():
|
||||
for path in paths:
|
||||
on_disk = (FIXTURES / source / path).read_text(encoding="utf-8")
|
||||
stored = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source, Document.path == path
|
||||
)
|
||||
)
|
||||
assert stored is not None, f"fixture doc {source}/{path} missing"
|
||||
assert stored.content == on_disk, f"content drifted for {source}/{path}"
|
||||
assert stored.summary == _expected_summary(on_disk, source, path), (
|
||||
f"summary is not the mock digest for {source}/{path}"
|
||||
)
|
||||
schunks = [c for c in stored.chunks if c.is_summary]
|
||||
assert len(schunks) == 1 and schunks[0].embedding is not None, (
|
||||
f"expected one embedded is_summary chunk for {source}/{path}"
|
||||
)
|
||||
|
||||
# ---- The fixture's token controls (the reproduction mechanism) ----
|
||||
ikea_text = (FIXTURES / IKEA_SOURCE / IKEA_DOC).read_text(encoding="utf-8")
|
||||
# (a) the name exists ONLY accented — no ASCII ``besta`` in the body,
|
||||
# so nothing but the phase-162 fold can bridge question ↔ doc;
|
||||
assert "BESTÅ" in ikea_text
|
||||
assert "besta" not in ikea_text.lower().replace("bestå", ""), ikea_text
|
||||
# (b) no ``build`` either: the accented question's only OTHER FTS
|
||||
# token — the small source must win on the NAME signal, never on
|
||||
# a content-word accident;
|
||||
assert "build" not in ikea_text.lower()
|
||||
# (c) the accented name sits PAST the 24-token digest window, so the
|
||||
# mock's ASCII-only digest never leaks a ``best`` lexeme the
|
||||
# pre-phase FTS could have hit (a real ``lite`` summary of the
|
||||
# live PDF never wrote the name in ASCII either).
|
||||
digest = TOKEN_RE.findall(ikea_text.lower())[:24]
|
||||
assert "best" not in digest and "besta" not in digest, digest
|
||||
|
||||
# ---- The engineered tiers (measured, stable across re-imports) ----
|
||||
_assert_tiers(ACCENTED_QUESTION, SUGGESTED_ACCENTED, RELATED_ACCENTED)
|
||||
_assert_tiers(ASCII_QUESTION, SUGGESTED_ASCII, RELATED_ASCII)
|
||||
_assert_tiers(FOLDER_QUESTION, SUGGESTED_FOLDER, RELATED_FOLDER)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped — the
|
||||
scripted turns never change it)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The pinned tiers (MEASURED with the app's real retrieval over the mock
|
||||
# embeddings — see the module docstring's before/after record)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: POST-FIX, accented question: the small source LEADS the tier (its
|
||||
#: name hit leads the lexical list ⇒ ``fuse`` rank 1 = 1/61, plus the
|
||||
#: per-document name-hit bonus 0.005), the five ASCII distractors
|
||||
#: follow in measured retrieval order, one of them falls to rank 6.
|
||||
SUGGESTED_ACCENTED = [
|
||||
IKEA,
|
||||
"phase_docs/140_pictorial_pdf_pages.md",
|
||||
"phase_docs/148_pdf_text_backfill.md",
|
||||
"phase_docs/152_seaweedfs_blob_ingest.md",
|
||||
"players_handbook/game_night_setup.md",
|
||||
]
|
||||
RELATED_ACCENTED = ["phase_docs/156_history_source_filter.md"]
|
||||
#: POST-FIX, ASCII question (the untouched phase-119 path): same doc,
|
||||
#: same #1 slot — the only difference is the measured order of two
|
||||
#: distractors that trade places on the extra ``besta`` FTS term.
|
||||
SUGGESTED_ASCII = [
|
||||
IKEA,
|
||||
"phase_docs/140_pictorial_pdf_pages.md",
|
||||
"phase_docs/148_pdf_text_backfill.md",
|
||||
"players_handbook/game_night_setup.md",
|
||||
"phase_docs/152_seaweedfs_blob_ingest.md",
|
||||
]
|
||||
RELATED_ASCII = ["phase_docs/156_history_source_filter.md"]
|
||||
#: The folder-map turn (the trigger's own tokens dilute the cosine to
|
||||
#: 0.296 — lexical support carries the gate) keeps the same tier.
|
||||
SUGGESTED_FOLDER = list(SUGGESTED_ACCENTED)
|
||||
RELATED_FOLDER = list(RELATED_ACCENTED)
|
||||
#: PRE-FIX (the measured miss, test 2): the folded token never reaches
|
||||
#: ``besta``, so the small source has ONE fused term (vector) against
|
||||
#: every distractor's TWO (vector + FTS ``build``) and the five
|
||||
#: distractors fill the tier — the IKEA doc is rank 6, i.e. the rank-6+
|
||||
#: ``related`` remainder. Exactly the live 2026-10-07 under-rank.
|
||||
PRE_SUGGESTED_ACCENTED = [
|
||||
"phase_docs/140_pictorial_pdf_pages.md",
|
||||
"phase_docs/148_pdf_text_backfill.md",
|
||||
"phase_docs/152_seaweedfs_blob_ingest.md",
|
||||
"players_handbook/game_night_setup.md",
|
||||
"phase_docs/156_history_source_filter.md",
|
||||
]
|
||||
PRE_RELATED_ACCENTED = [IKEA]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the phase-119 house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream (a
|
||||
#: response clone read in the background) — wire-level assertions on the
|
||||
#: ``done`` frame, independent of the rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict[str, Any]]:
|
||||
"""The SSE frames captured since the last submit, once the hook's
|
||||
background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _done(page: Page) -> dict[str, Any]:
|
||||
return next(f for f in _frames(page) if f.get("type") == "done")
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _last_query_log() -> QueryLog:
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(QueryLog)).all()
|
||||
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||
return rows[0]
|
||||
|
||||
|
||||
def _recorded_tiers(row: QueryLog) -> tuple[list[str], list[str]]:
|
||||
"""The durable record's tiers (LOCKED A3: the record is the suggested
|
||||
tier followed by the rank-6+ remainder — the wire's only faithful
|
||||
witness of the suggested set, since ``done.sources`` is the READ-DOCS
|
||||
citation surface and ``done.suggestions`` is the deflection chips)."""
|
||||
recorded = [s for s in row.sources.split(", ") if s]
|
||||
return recorded[:5], recorded[5:]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. (task 04 case 1) the accented question puts the 1-doc source in the
|
||||
# suggested top-5 — and it is no longer the rank-6 miss
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_accented_question_suggests_the_small_source(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""``How do I build the BESTÅ?`` grounds, and the suggested tier of
|
||||
the turn carries ``ikea_instructions/besta-frame-white__AA-1272068-
|
||||
11-100.md`` — SUGGESTED #1, measured. The suggested set is pinned on
|
||||
the wire through the durable record (LOCKED A3: ``sources`` = the
|
||||
suggested 5, then the rank-6+ remainder): the IKEA doc sits inside
|
||||
the first five entries, and the ``done`` frame's ``related`` list —
|
||||
the rank-6+ tier, the slot it occupied as the live miss — is EMPTY
|
||||
of it. A zero-read grounded turn chips nothing (phase 119 LOCKED
|
||||
A1); the answer renders grounded."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, ACCENTED_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = _done(page)
|
||||
assert done["deflected"] is False, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
# The miss was the IKEA doc landing in the rank-6+ remainder. It is
|
||||
# not there anymore — the tier it belongs to is the suggested one.
|
||||
related = [(s["source"], s["path"]) for s in done["related"]]
|
||||
assert (IKEA_SOURCE, IKEA_DOC) not in related, done["related"]
|
||||
# The ``suggestions`` field is the DEFLECTION contract (title chips
|
||||
# under a weak turn) — a grounded turn carries none, so it can never
|
||||
# be the tier witness (the durable record is, and is asserted below).
|
||||
assert done["suggestions"] == [], done["suggestions"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
# The de-emphasized rank-6+ row renders — and the doc it used to
|
||||
# hold (the miss) is a distractor now.
|
||||
row = _last_query_log()
|
||||
related_links = page.locator(".msg.brain .related-docs .related-doc")
|
||||
expect(related_links).to_have_count(len(RELATED_ACCENTED))
|
||||
for i, path in enumerate(RELATED_ACCENTED):
|
||||
expect(related_links.nth(i)).to_contain_text(path)
|
||||
expect(related_links.filter(has_text=IKEA)).to_have_count(0)
|
||||
|
||||
assert row.question == ACCENTED_QUESTION
|
||||
assert row.deflected is False
|
||||
suggested, related_record = _recorded_tiers(row)
|
||||
assert suggested == SUGGESTED_ACCENTED, row.sources
|
||||
assert related_record == RELATED_ACCENTED, row.sources
|
||||
assert suggested[0] == IKEA, row.sources
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. (task 04 case 1, the hypothesis) the under-rank is MEASURED in both
|
||||
# directions: pre-fix the doc loses the tier, post-fix it wins it
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_under_rank_is_measured_not_assumed(
|
||||
monkeypatch: pytest.MonkeyPatch, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The phase-162 hypothesis, measured rather than asserted: with the
|
||||
PRE-phase normalizer re-installed (``lower`` + alnum scan, no accent
|
||||
fold) the accented question degrades to the token ``best`` — no name
|
||||
hit — and the SAME retrieval over the SAME KB drops
|
||||
``ikea_instructions/besta-frame-white__AA-1272068-11-100.md`` out of
|
||||
the suggested top-5 into the rank-6+ remainder (the live miss,
|
||||
reproduced). With the shipped normalizer the folded ``besta`` token
|
||||
name-matches the filename and the doc is suggested #1. Re-running
|
||||
the measurement is the guard: if a future fixture or retriever
|
||||
change ever makes the pre-phase path win, this test fails and the
|
||||
recorded evidence is re-examined instead of trusted."""
|
||||
# The token side of the miss, measured.
|
||||
assert retriever.name_hit_tokens(ACCENTED_QUESTION) == ["build", "besta"]
|
||||
monkeypatch.setattr(retriever, "_normalize_name", _prefix_normalizer)
|
||||
assert retriever.name_hit_tokens(ACCENTED_QUESTION) == ["build", "best"]
|
||||
|
||||
pre_suggested, pre_related, _cosine, _fts = _tiers(ACCENTED_QUESTION)
|
||||
assert IKEA not in pre_suggested, (
|
||||
f"the pre-phase path WINS the tier — the fixture no longer "
|
||||
f"reproduces the miss: {pre_suggested}"
|
||||
)
|
||||
assert pre_suggested == PRE_SUGGESTED_ACCENTED, pre_suggested
|
||||
assert IKEA in pre_related, (
|
||||
f"the miss should surface as the rank-6+ remainder: {pre_related}"
|
||||
)
|
||||
assert pre_related == PRE_RELATED_ACCENTED, pre_related
|
||||
|
||||
monkeypatch.undo()
|
||||
# And the same measurement with the shipped fold: the doc is #1.
|
||||
post_suggested, post_related, _cosine, _fts = _tiers(ACCENTED_QUESTION)
|
||||
assert post_suggested == SUGGESTED_ACCENTED
|
||||
assert post_related == RELATED_ACCENTED
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. (task 04 case 2) the ASCII question keeps its phase-119 name hit
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ascii_question_still_name_matches(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""``How do I build a Besta?`` — the question the phase-119 rule
|
||||
already answered — still grounds on the same document (the fold must
|
||||
be behavior-preserving for the ASCII path: ``besta`` normalizes to
|
||||
``besta`` either way). Same tier position, same durable record,
|
||||
grounded, nothing chipped on a zero-read turn."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, ASCII_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = _done(page)
|
||||
assert done["deflected"] is False, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
expect(_last_brain(page).locator(".bubble")).to_contain_text(
|
||||
MOCK_ANSWER_MARKER, timeout=30_000
|
||||
)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == ASCII_QUESTION
|
||||
assert row.deflected is False
|
||||
suggested, related_record = _recorded_tiers(row)
|
||||
assert suggested == SUGGESTED_ASCII, row.sources
|
||||
assert suggested[0] == IKEA, row.sources
|
||||
assert related_record == RELATED_ASCII, row.sources
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. (task 04 case 3) the deterministic source inventory — the D2 tail
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_overview_tail_names_every_source_with_counts(
|
||||
synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The stored ``kb_overview`` row ENDS with the mechanical
|
||||
``Sources: …`` line — one entry per source, exact counts, singular
|
||||
``doc`` at k=1 — so the 1-document sources are visible to
|
||||
``<knowledge_base>`` whatever the lossy ``lite`` digest said (the
|
||||
2026-10-07 live finding: a 14-source index outlined as 3 groups).
|
||||
The tail is ONE line, appears exactly once, and sits after the
|
||||
model's outline (the model input never sees it)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(KbOverview)).all()
|
||||
assert len(rows) == 1, rows
|
||||
content = rows[0].content
|
||||
lines = content.splitlines()
|
||||
assert lines[-1] == INVENTORY, content
|
||||
assert content.count("Sources: ") == 1, content
|
||||
# The ``lite`` outline still leads (the tail is appended, never a
|
||||
# replacement).
|
||||
assert len(lines) >= 2, content
|
||||
assert lines[0].startswith("Knowledge base outline:"), content
|
||||
for source in SOURCE_DOCS:
|
||||
assert source in lines[-1], content
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 5. (task 04 case 4) the out-of-KB control — the gate verdict unchanged
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_out_of_kb_question_still_deflects(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""``What is the capital of Mongolia?`` matches nothing in this KB:
|
||||
the A8 verdict stays DEFLECTED (``deflected: true``, ``fts_hits ==
|
||||
0``, weak cosine under the mock-calibrated threshold), the answer
|
||||
carries no citation chips, and the only chips are the ``Maybe try``
|
||||
suggestions. The accent fold must never turn a real miss into a
|
||||
false grounding (the phase-119 battery's deflection sanity,
|
||||
localised)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, DEFLECT_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
done = _done(page)
|
||||
assert done["deflected"] is True, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
assert 2 <= len(done["suggestions"]) <= 3, done["suggestions"]
|
||||
|
||||
bubble = _last_brain(page)
|
||||
expect(bubble).to_have_class(re.compile(r"is-deflected"))
|
||||
expect(bubble.locator(".source-chip")).to_have_count(0)
|
||||
expect(
|
||||
bubble.locator(".maybe-try .suggestion-chip")
|
||||
).to_have_count(len(done["suggestions"]))
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == DEFLECT_QUESTION
|
||||
assert row.deflected is True
|
||||
assert row.fts_hits == 0, row.fts_hits
|
||||
assert row.top_score < 0.30, row.top_score
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 6. (task 04 case 3 / D3) the stored accented body stays grep-matchable
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_grep_folds_the_accented_body(synced_kb: None, db_ready: None) -> None:
|
||||
"""The content layer of the same miss: the stored body spells the
|
||||
product ``BESTÅ`` only, and the agent's ``grep`` tool folds BOTH the
|
||||
pattern and the line — so the ASCII pattern ``besta`` finds the
|
||||
accented line (the pre-fold tool returned nothing, which is why the
|
||||
live turn gave up on a document it had already opened), and the
|
||||
accented pattern still matches itself."""
|
||||
with SessionLocal() as db:
|
||||
stored = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == IKEA_SOURCE, Document.path == IKEA_DOC
|
||||
)
|
||||
)
|
||||
assert stored is not None
|
||||
assert "BESTÅ" in stored.content
|
||||
|
||||
ascii_hits = grep_document(stored.content, "besta")
|
||||
assert any("BESTÅ" in line for _no, line in ascii_hits), ascii_hits
|
||||
accented_hits = grep_document(stored.content, "BESTÅ")
|
||||
assert [line for _no, line in accented_hits] == [line for _no, line in ascii_hits]
|
||||
# A pattern that is not the name still finds nothing (the fold is an
|
||||
# accent bridge, not a wildcard).
|
||||
assert grep_document(stored.content, "bestaa") == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 7. the prompt-level witness — the 1-doc source's folder line rides the
|
||||
# HIGH prompt with the suggested tier
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_suggested_folder_lines_carry_the_small_source(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The accented question + the mock's ``repeat your folder map``
|
||||
trigger ⇒ the answer ends with the suggested-folder lines VERBATIM.
|
||||
The 1-doc source contributes ``ikea_instructions/:`` — a header with
|
||||
no sibling entries, because its only document IS the suggested doc —
|
||||
the visible proof the small source reached the prompt with the
|
||||
tier."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, FOLDER_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = _done(page)
|
||||
assert done["deflected"] is False, done
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(bubble).to_contain_text("ikea_instructions/:")
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.deflected is False
|
||||
suggested, _related = _recorded_tiers(row)
|
||||
assert suggested == SUGGESTED_FOLDER, row.sources
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# Assembly Instructions
|
||||
|
||||
Article number 127.206.11 — revision 4
|
||||
|
||||
Cam bolt 130618 · wall bracket 141413 · leveling foot 303154 · dowel 518041
|
||||
|
||||
## Package contents
|
||||
|
||||
- 1 x frame, white, 60 x 20 x 60 cm
|
||||
- 2 x side panels
|
||||
- 1 x back panel, hardboard
|
||||
- 12 x cam bolts with sleeve nuts
|
||||
- 1 bag of dowels
|
||||
|
||||
## Assembly
|
||||
|
||||
Insert sleeve nuts into the drilled holes, draw both side panels against the
|
||||
frame, tighten each cam bolt a quarter turn. Attach back panel with nails
|
||||
supplied, keeping it square and level. Fix wall bracket using plugs matched to
|
||||
wall material — furniture of this depth must always be secured.
|
||||
|
||||
## Product name
|
||||
|
||||
BESTÅ frame, white, 60 x 20 x 60 cm, AA 1272068 11 100. Wipe with a soft
|
||||
cloth dampened in water or a mild cleaner. Never soak the edges.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Pictorial PDF pages
|
||||
|
||||
How do I build a retrievable text layer for a scanned manual? Build one page
|
||||
image per PDF page, build an image description for each page, then build the
|
||||
index from those descriptions. The ASCII besta assembly PDF was the first
|
||||
manual to go through this path.
|
||||
|
||||
- build page images at 1400 px
|
||||
- build one description per page image
|
||||
- build chunks from descriptions, never from pixel guesses
|
||||
- besta manual: 24 scanned pages, 24 descriptions
|
||||
@@ -0,0 +1,9 @@
|
||||
# PDF text backfill
|
||||
|
||||
Backfill plan for scanned manuals. A manual that arrives without a text layer
|
||||
needs one before it can answer anything: run the pictorial page path, build
|
||||
descriptions for every page, keep the original page order.
|
||||
|
||||
- build: render, describe, index
|
||||
- besta shelf PDF: 24 scanned pages, no embedded text
|
||||
- question asked twice in review: how do I build this layer?
|
||||
@@ -0,0 +1,8 @@
|
||||
# SeaweedFS blob ingest
|
||||
|
||||
Object-store notes. Mount first, upload second.
|
||||
|
||||
- mount fs://manuals before the first upload
|
||||
- build the volume list, then copy the scan batch
|
||||
- besta scan batch: fs://manuals/2026-09/besta-scans.tar
|
||||
- checksum every archive after the copy
|
||||
@@ -0,0 +1,8 @@
|
||||
# History source filter
|
||||
|
||||
Turn history keeps the source each answer came from, so a filter can narrow
|
||||
the log to one source at a time.
|
||||
|
||||
- filter chip per source, sourced from turn history
|
||||
- a build log entry records the source it indexed
|
||||
- rerunning a question with a filter never rewrites history
|
||||
@@ -0,0 +1,7 @@
|
||||
# Game night setup
|
||||
|
||||
Room notes, games side of the shelf wall.
|
||||
|
||||
- besta unit holds box library; top shelf for dice trays
|
||||
- build tray insert before next session
|
||||
- keep rules binder beside clock, never stacked on besta top
|
||||
Vendored
+27
-1
@@ -1,4 +1,6 @@
|
||||
# Phase 119 retrieval battery (task 06, D6) — the 1,000-doc live replica.
|
||||
# Retrieval battery — phase 119 (task 06, D6) + phase 162 (task 04, D5),
|
||||
# run against the 1,000-doc live replica:
|
||||
# uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
|
||||
#
|
||||
# Gate (per the phase overview, post-phase measured numbers):
|
||||
# 1-3. each name question's suggested top-5 contains its canonical doc:
|
||||
@@ -10,8 +12,32 @@
|
||||
# 5. deflection sanity: the out-of-KB question's gate verdict + fts
|
||||
# count match the pre-phase replica run (HIGH, fts_hits=5) — the
|
||||
# name-hit change must not move it.
|
||||
#
|
||||
# Phase 162 appends the Besta pair (the 2026-10-07 live miss). Gate: the
|
||||
# suggested top-5 must contain the IKEA doc — ASCII filename
|
||||
# `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`, body that
|
||||
# spells the product `BESTÅ` only — for BOTH spellings: the accented one
|
||||
# needs the phase-162 question-side fold (D1) to bridge ASCII filename ←
|
||||
# accented question, the ASCII one is the phase-119 path (regression).
|
||||
# Either way ASCII `besta` documents must not outrank the doc that names
|
||||
# the product.
|
||||
#
|
||||
# SKIPPED LOUDLY on the local replica (the phase-140 precedent): the
|
||||
# 2026-09-05 snapshot carries NO `ikea_instructions`-class source —
|
||||
# `grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0, and
|
||||
# `SELECT DISTINCT source FROM documents` after the restore lists the
|
||||
# sources without it (verify, don't assume). The two questions still
|
||||
# run and must behave like any out-of-KB ask (no crash, no false
|
||||
# grounding on a doc that never named the product); the gate on the
|
||||
# rank itself can only close on a KB that has the source (owner's live
|
||||
# KB). The fixture that DOES reproduce the miss end-to-end is
|
||||
# tests/e2e/test_search_recall_small_sources.py over
|
||||
# tests/fixtures/recallkb (pre-fix ranks measured there, both
|
||||
# directions).
|
||||
How do I deploy gitea?
|
||||
How do I deploy forgejo?
|
||||
How do I set up the gateway?
|
||||
Do you have notes on the qwen 3.8 27b model?
|
||||
What is the capital of Mongolia?
|
||||
How do I build a Besta?
|
||||
How do I build the BESTÅ?
|
||||
|
||||
@@ -239,10 +239,11 @@ def test_changed_import_writes_overview_row(
|
||||
by_role = {m["role"]: m["content"] for m in llm.chat_calls[3]}
|
||||
assert "FOLDER_SUMMARY_MODE" in by_role["system"]
|
||||
assert by_role["user"].splitlines()[0] == "Folder: MyDocs"
|
||||
# The model's outline lands in the single row.
|
||||
# The model's outline lands in the single row, followed by the
|
||||
# phase-162 deterministic source inventory (LOCKED A2).
|
||||
row = _row(db)
|
||||
assert row is not None
|
||||
assert row.content == "Summary of MyDocs"
|
||||
assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)"
|
||||
assert row.updated_at is not None
|
||||
# The phase's required log line (PLAN §9).
|
||||
assert any("overview: regenerated docs=2 chars=" in r.getMessage() for r in records)
|
||||
@@ -274,7 +275,8 @@ def test_unchanged_reimport_does_not_call_lite(
|
||||
)
|
||||
assert len(llm.chat_calls) == 4 # no new lite call
|
||||
row = _row(db)
|
||||
assert row is not None and row.content == "Summary of MyDocs"
|
||||
assert row is not None
|
||||
assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)"
|
||||
assert _version(db) == 1 # phase 53: an unchanged re-run never bumps
|
||||
|
||||
|
||||
@@ -345,7 +347,8 @@ def test_limit_run_skips_overview(
|
||||
# lite call; the overview + folder gates skip under --limit.
|
||||
assert len(llm.chat_calls) == 5
|
||||
row = _row(db)
|
||||
assert row is not None and row.content == "Summary of MyDocs"
|
||||
assert row is not None
|
||||
assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)"
|
||||
assert _version(db) == 1 # phase 53: --limit debug runs never bump
|
||||
|
||||
|
||||
@@ -462,5 +465,8 @@ def test_overview_outline_names_the_newest_docs_when_the_cap_bites(
|
||||
# would have been "mydocs old 01 md old 01 summary of" (the oldest
|
||||
# batch) — the "summaries only up to phase 113" shape this pin
|
||||
# forbids.
|
||||
assert row.content == "Knowledge base outline:\n- mydocs zz new 03 md zz new 03"
|
||||
assert row.content == (
|
||||
"Knowledge base outline:\n- mydocs zz new 03 md zz new 03\n"
|
||||
"Sources: MyDocs (6 docs)"
|
||||
)
|
||||
assert "old" not in row.content # no oldest-batch token may survive the cut
|
||||
|
||||
@@ -367,7 +367,8 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
# teaching no-match line instead of a trusted miss.
|
||||
assert grep["description"] == (
|
||||
"Search the indexed documents for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines "
|
||||
"(case- and accent-insensitive — accents fold: `besta` "
|
||||
"matches `BESTÅ`) and return up to 20 matching lines "
|
||||
"as `source/path:line: text` — a locator, not a "
|
||||
"context-adder: read the winner with `read` — for a "
|
||||
"large document, use the reported line number as "
|
||||
@@ -387,8 +388,11 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
assert set(grep_params["properties"]) == {"pattern", "path"}
|
||||
assert all(p["type"] == "string" for p in grep_params["properties"].values())
|
||||
assert grep_params["properties"]["pattern"]["description"] == (
|
||||
"The exact text to search for (a plain substring, "
|
||||
"not a regex — no '.*', no '\\.', no character classes)"
|
||||
"The exact text to search for (a plain "
|
||||
"substring — case- and accent-insensitive, "
|
||||
"accents fold: `besta` matches `BESTÅ` — "
|
||||
"not a regex — no '.*', no "
|
||||
"'\\.', no character classes)"
|
||||
)
|
||||
# Phase 72 (task 02): the bare-path contract is stated up front;
|
||||
# task 05 (live gate iterations 1-8): the one-known-document clause
|
||||
@@ -4000,6 +4004,27 @@ def test_origin_lines_image_doc_keeps_marker_position(
|
||||
# ---------- grep (the phase-68 A5 contract under the new name) ----------
|
||||
|
||||
|
||||
def test_grep_document_folds_accents_both_ways() -> None:
|
||||
"""Phase 162 (LOCKED A3): grep folds BOTH the pattern and each line
|
||||
through the shared ``app.rag.retriever._fold`` plus lowercase — an
|
||||
ASCII pattern hits accented content and an accented pattern hits
|
||||
ASCII content, one fold, never drifting from the name-hit path."""
|
||||
# ASCII pattern, accented content.
|
||||
assert agent.grep_document("The BESTÅ frame (AA-1272068-11-100)", "besta") == [
|
||||
(1, "The BESTÅ frame (AA-1272068-11-100)")
|
||||
]
|
||||
assert agent.grep_document("line one\nBÅsta shelf\nline three", "basta") == [
|
||||
(2, "BÅsta shelf")
|
||||
]
|
||||
assert agent.grep_document("café-au-lait", "cafe") == [(1, "café-au-lait")]
|
||||
# Accented pattern, ASCII content — the pattern's own accents fold.
|
||||
assert agent.grep_document("The besta frame", "BESTÅ") == [(1, "The besta frame")]
|
||||
# A genuinely non-matching folded pattern still returns no lines.
|
||||
assert agent.grep_document("The besta frame", "bestb") == []
|
||||
# The fold bridges Ä/Ö/Ü/ß too (the same _fold as name matching).
|
||||
assert agent.grep_document("Straße", "strasse") == [(1, "Straße")]
|
||||
|
||||
|
||||
def test_grep_document_case_insensitive_line_numbers() -> None:
|
||||
"""Case-insensitive fixed substring, 1-based line numbers, file order,
|
||||
repeated matches within a line collapse to one match (grep semantics)."""
|
||||
|
||||
+144
-6
@@ -5,7 +5,11 @@ per-document user lines (source/path/title/first summary line), and the
|
||||
input cap with the shared ``[…truncated…]`` marker. The loader and
|
||||
regenerator tests run against the local compose Postgres (preferred —
|
||||
real single-row upsert), skipping with clear instructions when the stack
|
||||
is not up — same pattern as ``test_importer.py``.
|
||||
is not up — same pattern as ``test_importer.py``. The phase-162 section
|
||||
pins the deterministic ``Sources: …`` inventory tail (LOCKED A2): the
|
||||
format/order/counts, the 1-doc survival, the happy-path stored content,
|
||||
the byte-identical zero-doc / ``LLMError`` paths, and the byte-identical
|
||||
model input.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -15,7 +19,7 @@ import uuid
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
@@ -25,6 +29,7 @@ from app.rag.overview import (
|
||||
KB_OVERVIEW_MODE,
|
||||
OVERVIEW_INSTRUCTION,
|
||||
SYSTEM_PROMPT,
|
||||
_source_inventory,
|
||||
build_overview_prompt,
|
||||
load_kb_overview,
|
||||
regenerate_overview,
|
||||
@@ -302,11 +307,13 @@ def test_regenerate_happy_path_creates_single_row_id_1(
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None, "the single row must be upserted"
|
||||
assert row.id == 1
|
||||
assert row.content == REPLY
|
||||
# Phase 162 (LOCKED A2): the stored content carries the
|
||||
# deterministic source-inventory tail after the model text.
|
||||
assert row.content == REPLY + "\nSources: Homelab (2 docs)"
|
||||
assert row.updated_at is not None
|
||||
age = datetime.now(UTC) - row.updated_at
|
||||
assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp"
|
||||
assert f"overview: regenerated docs=2 chars={len(REPLY)}" in caplog.text
|
||||
assert f"overview: regenerated docs=2 chars={len(row.content)}" in caplog.text
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
@@ -324,7 +331,7 @@ def test_regenerate_updates_existing_row_in_place(db: Session, clean_overview) -
|
||||
assert count == 1, "still exactly one row after the re-generation"
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None
|
||||
assert row.content == REPLY # replaced, not appended
|
||||
assert row.content == REPLY + "\nSources: Homelab (1 doc)" # replaced, not appended
|
||||
assert row.updated_at is not None and row.updated_at > past
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
@@ -511,4 +518,135 @@ def test_regenerate_opens_own_session_when_none(db: Session, clean_overview) ->
|
||||
|
||||
assert ok is True
|
||||
row = db.get(KbOverview, 1) # fresh lookup via the test session
|
||||
assert row is not None and row.content == REPLY
|
||||
assert row is not None and row.content == REPLY + "\nSources: Homelab (1 doc)"
|
||||
|
||||
|
||||
# ---------- phase 162: the deterministic source inventory (LOCKED A2) -------
|
||||
|
||||
|
||||
def test_source_inventory_format_counts_and_catalog_order() -> None:
|
||||
"""Exact format, singular/plural, and catalog order: one entry per
|
||||
DISTINCT source — ``<name> (<k> doc|docs)`` joined with ``, ``
|
||||
behind ``Sources: ``, ordered ``source, path`` (the scrambled
|
||||
input rows prove the catalog order is the OUTPUT order, not the
|
||||
recency-first order of the rows the generator reads)."""
|
||||
rows = [
|
||||
("b", "x.md", "X", None),
|
||||
("a", "y.md", "Y", None),
|
||||
("c", "p1.md", "P1", None),
|
||||
("b", "z.md", "Z", None),
|
||||
("c", "p2.md", "P2", None),
|
||||
("c", "p3.md", "P3", None),
|
||||
]
|
||||
assert _source_inventory(rows) == "Sources: a (1 doc), b (2 docs), c (3 docs)"
|
||||
|
||||
|
||||
def test_source_inventory_keeps_a_one_doc_source_visible() -> None:
|
||||
"""The 2026-10-07 ``ikea_instructions`` case: a 1-doc source among
|
||||
many appears with ``(1 doc)`` — no source is invisible to
|
||||
``<knowledge_base>``, whatever the ``lite`` outline says."""
|
||||
rows = [
|
||||
("gufo", "a.md", "A", None),
|
||||
(
|
||||
"ikea_instructions",
|
||||
"besta-frame-white__AA-1272068-11-100.md",
|
||||
"Assembly Instructions",
|
||||
None,
|
||||
),
|
||||
("gufo", "b.md", "B", None),
|
||||
("players_handbook", "core.md", "Core", None),
|
||||
]
|
||||
assert _source_inventory(rows) == (
|
||||
"Sources: gufo (2 docs), ikea_instructions (1 doc), "
|
||||
"players_handbook (1 doc)"
|
||||
)
|
||||
|
||||
|
||||
def test_regenerate_appends_the_inventory_tail_to_the_stored_content(
|
||||
db: Session, clean_overview, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Happy path, exact pin on the fake's fixed output: the stored row
|
||||
= model text (trailing newlines rstripped) + ``\\n`` + the inventory
|
||||
line; the log's chars = the FULL stored content, tail included."""
|
||||
_truncate_documents(db)
|
||||
try:
|
||||
_add_doc(db, "players_handbook", "rpg.md", "RPG")
|
||||
_add_doc(db, "Homelab", "b.md", "B")
|
||||
_add_doc(db, "Homelab", "a.md", "A")
|
||||
llm = _FakeLLM(reply=REPLY + "\n\n")
|
||||
expected = REPLY + "\nSources: Homelab (2 docs), players_handbook (1 doc)"
|
||||
with caplog.at_level(logging.INFO, logger="app.rag.overview"):
|
||||
ok = asyncio.run(regenerate_overview(llm, db))
|
||||
assert ok is True
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None
|
||||
assert row.content == expected # no blank line before the tail
|
||||
assert f"overview: regenerated docs=3 chars={len(expected)}" in caplog.text
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
|
||||
def test_regenerate_model_input_never_gains_the_inventory_tail(
|
||||
db: Session, clean_overview
|
||||
) -> None:
|
||||
"""The model input is byte-identical to the pre-phase prompt: the
|
||||
user message the fake received equals ``build_overview_prompt`` for
|
||||
the same catalogue rows — the tail is never sent to ``lite``."""
|
||||
_truncate_documents(db)
|
||||
try:
|
||||
_add_doc(db, "Homelab", "a.md", "A", "A summary.\nSource: Homelab/a.md")
|
||||
_add_doc(db, "Deployments", "b.yaml", "B")
|
||||
llm = _FakeLLM()
|
||||
assert asyncio.run(regenerate_overview(llm, db)) is True
|
||||
rows = db.execute(
|
||||
select(Document.source, Document.path, Document.title, Document.summary)
|
||||
.order_by(Document.created_at.desc(), Document.source, Document.path)
|
||||
).all()
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
user = next(m["content"] for m in llm.messages if m["role"] == "user")
|
||||
_, expected_user = build_overview_prompt([tuple(r) for r in rows])
|
||||
assert user == expected_user
|
||||
assert "Sources:" not in user
|
||||
|
||||
|
||||
def test_regenerate_zero_docs_keeps_tailless_pre_phase_row(
|
||||
db: Session, clean_overview
|
||||
) -> None:
|
||||
"""Zero-doc early return: byte-identical pre-phase behavior — the
|
||||
pre-phase (tail-less) row survives untouched, no tail is invented."""
|
||||
db.execute(text("DELETE FROM kb_overview"))
|
||||
pre_phase_row = "- Old outline\n- from before phase 162"
|
||||
db.add(KbOverview(id=1, content=pre_phase_row))
|
||||
db.commit()
|
||||
_truncate_documents(db)
|
||||
llm = _FakeLLM()
|
||||
ok = asyncio.run(regenerate_overview(llm, db))
|
||||
assert ok is False
|
||||
assert llm.calls == 0
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None and row.content == pre_phase_row
|
||||
assert "Sources:" not in row.content
|
||||
|
||||
|
||||
def test_regenerate_llm_error_preserves_an_existing_inventory_tail(
|
||||
db: Session, clean_overview
|
||||
) -> None:
|
||||
"""The tail is CONTENT: a previous row that already carried a
|
||||
``Sources: …`` tail survives an ``LLMError`` byte-identically —
|
||||
including the tail (an old outline is better than none)."""
|
||||
db.execute(text("DELETE FROM kb_overview"))
|
||||
previous = "- Old outline\nSources: Homelab (3 docs)"
|
||||
db.add(KbOverview(id=1, content=previous))
|
||||
db.commit()
|
||||
_truncate_documents(db)
|
||||
try:
|
||||
_add_doc(db, "Homelab", "a.md", "A")
|
||||
llm = _FakeLLM(fail=LLMError("simulated lite-model failure"))
|
||||
ok = asyncio.run(regenerate_overview(llm, db))
|
||||
assert ok is False
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None and row.content == previous
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
@@ -17,6 +17,7 @@ import hashlib
|
||||
from app.rag.prompts import (
|
||||
ORIGIN_RULE,
|
||||
PERSONA,
|
||||
SEARCH_DISCIPLINE_RULE,
|
||||
SUGGEST_INTRO,
|
||||
TOOLS_SECTION,
|
||||
_base,
|
||||
@@ -87,9 +88,34 @@ def test_high_and_low_bases_byte_locked() -> None:
|
||||
#: inserted — the prefix (the ``ls``-clause opening) and the suffix
|
||||
#: (the discipline-rules ending) survived byte-identical, so they are
|
||||
#: the same anchors as pre-phase-145; the pre-phase-145 sha/length are
|
||||
#: kept below for the diff-style pin.
|
||||
TOOLS_SECTION_SHA256 = "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
|
||||
TOOLS_SECTION_LEN = 3279
|
||||
#: kept below for the chained diff-style pin. RE-CUT again for phase 162
|
||||
#: (task 03, LOCKED A3+A4): the ``grep`` sentence's ``(case-insensitive)``
|
||||
#: parenthetical widened to the folded contract ``(case- and
|
||||
#: accent-insensitive — accents fold: besta matches BESTÅ)``, and the
|
||||
#: LOCKED search-discipline sentence (:data:`SEARCH_DISCIPLINE_RULE`) was
|
||||
#: inserted immediately after the grep sentence (before the ``For a normal
|
||||
#: search pass`` clause). Only those two byte-groups moved — the prefix
|
||||
#: (the ``ls``-clause opening) and the suffix (the discipline-rules
|
||||
#: ending) survive byte-identical; the pre-phase-162 sha/length are kept
|
||||
#: below for the chained diff-style pin.
|
||||
TOOLS_SECTION_SHA256 = "988642f9dc6bb2a4a1d44a83b5368ff4cdb9df4e181b2fe3dc2fb241b1857595"
|
||||
TOOLS_SECTION_LEN = 3818
|
||||
|
||||
# The pre-phase-162 anchors — the section BEFORE phase 162's grep-fold +
|
||||
# search-discipline insertion. Kept for the chained diff-style pin.
|
||||
TOOLS_SECTION_PRE_PHASE_162_SHA256 = (
|
||||
"41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
|
||||
)
|
||||
TOOLS_SECTION_PRE_PHASE_162_LEN = 3279
|
||||
|
||||
# The grep contract parenthetical exactly as it changed in phase 162
|
||||
# (LOCKED A3): the pre-phase ``(case-insensitive)`` vs the folded one now
|
||||
# embedded in ``TOOLS_SECTION``. Used to revert phase 162 for the diff pin.
|
||||
_GREP_CONTRACT_PRE_PHASE_162 = "(case-insensitive)"
|
||||
_GREP_CONTRACT_PHASE_162 = (
|
||||
"(case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)"
|
||||
)
|
||||
|
||||
TOOLS_SECTION_PREFIX = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
@@ -159,19 +185,46 @@ def test_tools_section_origin_rule_verbatim_at_its_position() -> None:
|
||||
) in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_origin_rule_is_the_only_byte_change() -> None:
|
||||
"""Phase 145 (task 02) diff-style pin: the section's OTHER bytes are
|
||||
byte-identical to pre-phase — the pre-phase section with ONLY the new
|
||||
sentence-group inserted. Removing the rule (and the single space
|
||||
that now separates it from the ``grep`` clause) reproduces the
|
||||
pre-phase-145 section byte-for-byte (sha + length), and the length
|
||||
delta is exactly the rule's length plus that one inserted space."""
|
||||
assert len(TOOLS_SECTION) == (
|
||||
TOOLS_SECTION_PRE_PHASE_145_LEN + len(ORIGIN_RULE) + 1
|
||||
)
|
||||
pre_phase = TOOLS_SECTION.replace(ORIGIN_RULE + " ", "")
|
||||
assert len(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_LEN
|
||||
assert _sha256(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_SHA256
|
||||
def test_tools_section_search_discipline_rule_verbatim_at_its_position() -> None:
|
||||
"""Phase 162 (task 03, LOCKED A4): the search-discipline rule is
|
||||
LOCKED verbatim and sits at its EXACT position — immediately after
|
||||
the grep sentence (the ``…retry with the plain text you expect to
|
||||
see.`` clause) and BEFORE the ``For a normal search pass`` clause.
|
||||
The rule + its exact boundaries are pinned as one contiguous span, so
|
||||
a move or re-spelling of ANY byte fails."""
|
||||
assert SEARCH_DISCIPLINE_RULE in TOOLS_SECTION
|
||||
assert TOOLS_SECTION.count(SEARCH_DISCIPLINE_RULE) == 1 # exactly once
|
||||
assert (
|
||||
"retry with the plain text you expect to see. "
|
||||
+ SEARCH_DISCIPLINE_RULE
|
||||
+ " For a normal search pass only `pattern`"
|
||||
) in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_grep_fold_contract_present() -> None:
|
||||
"""Phase 162 (task 03, LOCKED A3): the grep sentence now states the
|
||||
FOLDED contract — the ``(case-insensitive)`` parenthetical is replaced
|
||||
byte-for-byte by the folded one, and the old one is gone."""
|
||||
assert _GREP_CONTRACT_PHASE_162 in TOOLS_SECTION
|
||||
assert _GREP_CONTRACT_PRE_PHASE_162 not in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_phase162_and_145_are_the_only_byte_changes() -> None:
|
||||
"""Chained diff-style pin: the ONLY byte changes are phase 162's (the
|
||||
folded grep contract + :data:`SEARCH_DISCIPLINE_RULE`) and phase 145's
|
||||
(:data:`ORIGIN_RULE`). Reverting phase 162 (drop the rule + its
|
||||
separating space, restore the pre-phase grep parenthetical) reproduces
|
||||
the pre-phase-162 section byte-for-byte; reverting the origin rule on
|
||||
top of that reproduces the pre-phase-145 section byte-for-byte."""
|
||||
# Revert phase 162 -> pre-phase-162.
|
||||
pre_162 = TOOLS_SECTION.replace(SEARCH_DISCIPLINE_RULE + " ", "")
|
||||
pre_162 = pre_162.replace(_GREP_CONTRACT_PHASE_162, _GREP_CONTRACT_PRE_PHASE_162)
|
||||
assert len(pre_162) == TOOLS_SECTION_PRE_PHASE_162_LEN
|
||||
assert _sha256(pre_162) == TOOLS_SECTION_PRE_PHASE_162_SHA256
|
||||
# Revert phase 145 -> pre-phase-145.
|
||||
pre_145 = pre_162.replace(ORIGIN_RULE + " ", "")
|
||||
assert len(pre_145) == TOOLS_SECTION_PRE_PHASE_145_LEN
|
||||
assert _sha256(pre_145) == TOOLS_SECTION_PRE_PHASE_145_SHA256
|
||||
|
||||
|
||||
# ---------- the LOW prompt's locked DEFLECT_MODE body ----------
|
||||
|
||||
+19
-10
@@ -50,6 +50,7 @@ from app.models import Document
|
||||
from app.rag.prompts import (
|
||||
ORIGIN_RULE,
|
||||
PERSONA,
|
||||
SEARCH_DISCIPLINE_RULE,
|
||||
SHOW_IMAGE_CLAUSE,
|
||||
SUGGEST_INTRO,
|
||||
TOOLS_SECTION,
|
||||
@@ -286,14 +287,14 @@ def test_tools_section_markers_and_new_tool_names() -> None:
|
||||
|
||||
def test_tools_section_teaches_the_harness_shapes() -> None:
|
||||
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
|
||||
case-insensitive exact-string locator contract (up to 20
|
||||
case- and accent-insensitive exact-string locator contract (up to 20
|
||||
``source/path:line: text`` lines, a locator not a context-adder),
|
||||
and ``read``'s combined ``source/path`` + full content. (Phase 72:
|
||||
the source-name scope clause and the combined-identity clause are
|
||||
pinned byte-for-byte in
|
||||
:func:`test_tools_section_phase72_contract_clauses`.)"""
|
||||
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
|
||||
assert "case-insensitive" in TOOLS_SECTION
|
||||
assert "case- and accent-insensitive" in TOOLS_SECTION # phase 162 fold
|
||||
assert "up to 20" in TOOLS_SECTION
|
||||
assert "source/path:line: text" in TOOLS_SECTION
|
||||
assert "locator, not a context-adder" in TOOLS_SECTION
|
||||
@@ -414,20 +415,28 @@ def test_tools_section_phase118_summary_seed_read_clause() -> None:
|
||||
"| path: Y | title: Z`; to find one specific document without "
|
||||
"listing, use `grep`."
|
||||
) in TOOLS_SECTION
|
||||
# The grep clause — byte-identical.
|
||||
# The grep clause — folded contract (phase 162, LOCKED A3), up to the
|
||||
# search-discipline sentence, byte-identical.
|
||||
assert (
|
||||
"`grep` locates an exact string (case-insensitive) in the "
|
||||
"`grep` locates an exact string (case- and accent-insensitive — "
|
||||
"accents fold: `besta` matches `BESTÅ`) in the "
|
||||
"indexed documents and returns up to 20 matching "
|
||||
"`source/path:line: text` lines — a locator, not a "
|
||||
"context-adder: read the winner with `read`. A grep pattern "
|
||||
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
|
||||
"literal text there; if such a pattern returns no matches, "
|
||||
"retry with the plain text you expect to see. For a normal "
|
||||
"search pass only `pattern` — its optional `path` argument "
|
||||
"limits the search to one document you already know, by the "
|
||||
"same combined `source/path` string; never a source name — a "
|
||||
"bare document path (without the source name) will not "
|
||||
"resolve there either."
|
||||
"retry with the plain text you expect to see. "
|
||||
) in TOOLS_SECTION
|
||||
# The LOCKED search-discipline sentence sits immediately after the
|
||||
# grep sentence, before the path clause (phase 162, LOCKED A4).
|
||||
assert (
|
||||
"retry with the plain text you expect to see. "
|
||||
+ SEARCH_DISCIPLINE_RULE
|
||||
+ " For a normal search pass only `pattern` — its optional "
|
||||
"`path` argument limits the search to one document you already "
|
||||
"know, by the same combined `source/path` string; never a "
|
||||
"source name — a bare document path (without the source name) "
|
||||
"will not resolve there either."
|
||||
) in TOOLS_SECTION
|
||||
# The discipline rules — byte-identical.
|
||||
assert (
|
||||
|
||||
@@ -1346,3 +1346,156 @@ def test_bonus_lives_in_the_selection_layer_only() -> None:
|
||||
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
|
||||
}
|
||||
assert before == after
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 162, D1 — the accent-folded name normalization (LOCKED A1): the
|
||||
# 2026-10-07 live gap, the ASCII filename vs the accented content layer
|
||||
# (``besta`` <-> ``BESTÅ``, both directions, ONE function)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from app.rag.retriever import _fold, _name_parts # noqa: E402
|
||||
|
||||
|
||||
def test_fold_accent_ligature_and_sharp_s_cases() -> None:
|
||||
"""The shared accent fold (phase 162, LOCKED A1): NFKD decomposition,
|
||||
combining marks dropped — ``Å/å → a``, ``Ä/Ö/Ü → a/o/u`` (the letter
|
||||
survives, the mark drops), ``ß → ss`` (the Unicode fold —
|
||||
``str.casefold``; NFKD alone leaves ``ß`` intact), the ``fi`` ligature
|
||||
``→ fi``, ``ç → c``."""
|
||||
assert _fold("BESTÅ") == "besta"
|
||||
assert _fold("Åsa") == "asa"
|
||||
assert _fold("ÄÖÜäöü") == "aouaou"
|
||||
assert _fold("Straße") == "strasse"
|
||||
assert _fold("fine") == "fine"
|
||||
assert _fold("garçon") == "garcon"
|
||||
|
||||
|
||||
def test_fold_survives_non_decomposing_characters() -> None:
|
||||
"""Characters with NO decomposition (CJK) survive the fold unchanged —
|
||||
the caller's alnum scan drops them, exactly as pre-phase. Compat-only
|
||||
characters (``①``) NFKD to their ASCII base (``1``); that base is
|
||||
below :data:`NAME_TOKEN_MIN_LEN`, so it never becomes a name token
|
||||
(the name-signal behavior for such characters is unchanged)."""
|
||||
assert _fold("漢字") == "漢字"
|
||||
assert _fold("①") == "1"
|
||||
|
||||
|
||||
def test_normalize_name_is_accent_folded() -> None:
|
||||
"""``_normalize_name`` = the alnum scan of ``_fold(s.lower())`` —
|
||||
``Qwen 3.8 → qwen38`` UNCHANGED, ``BESTÅ → besta`` NEW (pre-phase it
|
||||
was ``best``); non-folding characters still drop out (``漢字 → ""``)."""
|
||||
assert _normalize_name("Qwen 3.8") == "qwen38"
|
||||
assert _normalize_name("BESTÅ") == "besta"
|
||||
assert _normalize_name("BÅsta") == "basta"
|
||||
assert _normalize_name("straße") == "strasse"
|
||||
assert _normalize_name("fine") == "fine"
|
||||
assert _normalize_name("garçon") == "garcon"
|
||||
assert _normalize_name("漢字") == ""
|
||||
# Compat-only base survives (``①`` → ``1``) but is a 1-char token —
|
||||
# below the minimum, never a name candidate (see the token test).
|
||||
assert _normalize_name("①") == "1"
|
||||
|
||||
|
||||
def test_name_hit_tokens_accented_question_bridges() -> None:
|
||||
"""The question side of the bridge: the 2026-10-07 battery question
|
||||
produces the ``besta`` token (pre-phase: ``best`` — the exact list is
|
||||
pinned; ``how``/``do``/``i``/``the`` are below NAME_TOKEN_MIN_LEN).
|
||||
The fold covers every pinned case: ``BESTÅ``, umlauts, ``ß``, the
|
||||
``fi`` ligature, ``ç``."""
|
||||
assert name_hit_tokens("How do I build the BESTÅ?") == ["build", "besta"]
|
||||
assert name_hit_tokens("how do I fine the Über strasse") == [
|
||||
"fine", "uber", "strasse",
|
||||
]
|
||||
assert name_hit_tokens("MÖRK ÖVER BÄSTA GARÇON") == [
|
||||
"mork", "over", "basta", "garcon",
|
||||
]
|
||||
|
||||
|
||||
def test_name_hit_tokens_min_length_on_the_folded_form() -> None:
|
||||
"""NAME_TOKEN_MIN_LEN is measured on the FOLDED form (``BÄST`` →
|
||||
``bast`` (4, kept) while ``BÄ`` → ``ba`` (2, dropped)), and
|
||||
non-folding tokens still drop out: ``漢字`` normalizes to ``""`` and
|
||||
``①`` to the 1-char ``1`` — neither can ever be a candidate. The
|
||||
``1`` is compat-decomposition (NFKD — LOCKED A1): the ``①`` token
|
||||
itself never survives, and the only footprint a digit-base can leave
|
||||
is a versioned-JOIN artifact (``the`` + ``1`` → ``the1``) whose sole
|
||||
match route is PREFIXING a normalized path part — no real path does,
|
||||
so the name signal for such characters is unchanged."""
|
||||
assert name_hit_tokens("BÄST BÄ") == ["bast"]
|
||||
assert name_hit_tokens("tell me about it ① 漢字") == ["tell", "about"]
|
||||
# The compat-decomposed ``1`` acts as a version token for the join —
|
||||
# pinned as the real behavior (a prefix-class candidate no path hits).
|
||||
assert name_hit_tokens("tell me about the ① 漢字") == [
|
||||
"tell", "about", "the1",
|
||||
]
|
||||
|
||||
|
||||
def test_name_parts_fold_both_the_ascii_and_accented_stem() -> None:
|
||||
"""The path side of the bridge: the ASCII IKEA filename yields
|
||||
``besta`` in the EQUAL set (the stem sub-component — the phase-119
|
||||
A2 case, unchanged), and the accented filename ``BÅsta.md`` yields
|
||||
``basta`` (the folded stem — already in the equal set via the
|
||||
normalized-stem member, no _name_parts code change)."""
|
||||
_prefix, equal = _name_parts(
|
||||
"ikea/besta-frame-white__AA-1272068-11-100.pdf"
|
||||
)
|
||||
assert "besta" in equal # sub-component, unchanged
|
||||
_prefix2, equal2 = _name_parts("manuals/BÅsta.md")
|
||||
assert "basta" in equal2 # folded stem — an ASCII question matches it
|
||||
|
||||
|
||||
def test_name_hit_chunks_accented_question_hits_ascii_doc() -> None:
|
||||
"""The bridge, end to end (direction 1): the accented question
|
||||
``BESTÅ`` name-hits the ASCII-stem document — the token ``besta``
|
||||
EQUALS the stem sub-component of
|
||||
``besta-frame-white__AA-1272068-11-100`` (the 2026-10-07 live miss;
|
||||
pre-phase the token was ``best`` and the hit was impossible)."""
|
||||
ikea = _doc(
|
||||
"ikea/besta-frame-white__AA-1272068-11-100.pdf",
|
||||
"MONTERING BÄSTA",
|
||||
source="ikea_instructions",
|
||||
)
|
||||
distractor = _doc("notes/meeting.md", "we discussed besta", source="Brain-of-Reese")
|
||||
session = _FakeSession(
|
||||
[_name_row(ikea), _name_row(distractor)],
|
||||
[_name_hit_lateral_row(ikea)],
|
||||
)
|
||||
out = _name_hit_chunks(session, "How do I build the BESTÅ?") # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [ikea.path]
|
||||
assert out[0].name_hit is True
|
||||
assert out[0].fts_hit is True
|
||||
assert out[0].cosine == 0.0
|
||||
|
||||
|
||||
def test_name_hit_chunks_ascii_question_hits_accented_doc() -> None:
|
||||
"""The bridge, end to end (direction 2): the ASCII question token
|
||||
``basta`` name-hits the ACCENTED filename ``BÅsta.md`` — the folded
|
||||
stem enters the equal set as ``basta`` (pre-phase the accented stem
|
||||
normalized to ``bsta`` — the ``å`` simply dropped out of the alnum
|
||||
scan — so the match was impossible)."""
|
||||
accented = _doc("manuals/BÅsta.md", "MONTERING", source="ikea_instructions")
|
||||
miss = _doc("manuals/hesta.md", "MONTERING", source="ikea_instructions")
|
||||
session = _FakeSession(
|
||||
[_name_row(accented), _name_row(miss)],
|
||||
[_name_hit_lateral_row(accented)],
|
||||
)
|
||||
out = _name_hit_chunks(session, "How do I assemble the basta shelf?") # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [accented.path]
|
||||
assert out[0].name_hit is True
|
||||
|
||||
|
||||
def test_name_hit_chunks_fold_preserves_digitless_precision() -> None:
|
||||
"""The fold does NOT relax the A2 match classes: the accented token
|
||||
``déploy`` folds to ``deploy`` and STILL does not hit the
|
||||
``Deployments/`` document (the part normalizes to ``deployments`` ≠
|
||||
``deploy`` — titles are never matched, digits never enter); only the
|
||||
exact ``gitea`` component hits."""
|
||||
precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
|
||||
gitea = _doc("notes/gitea.md", "body")
|
||||
session = _FakeSession(
|
||||
[_name_row(precision), _name_row(gitea)],
|
||||
[_name_hit_lateral_row(gitea)],
|
||||
)
|
||||
out = _name_hit_chunks(session, "how do I déploy gitea") # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [gitea.path]
|
||||
|
||||
Reference in New Issue
Block a user