phase: 162_search_recall_small_sources
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)** - Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched. - Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated. - Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched). - Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`). - Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382. - `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**. - `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**. - Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness. - Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB. - Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.) - Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass. - Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`". - Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
This commit is contained in:
@@ -280,7 +280,12 @@ reflect the new files.
|
||||
- A run that changed the KB also regenerates the **KB overview** and the
|
||||
**folder summaries** (the drill-down catalog's stored per-folder
|
||||
descriptions) with the `lite` model — best-effort: a summary failure
|
||||
never fails the import. Because the sync only rewrites the folders a
|
||||
never fails the import. The stored overview ends with a deterministic
|
||||
`Sources: <name> (<k> doc|docs), …` line — every source with its exact
|
||||
document count, catalog order, computed without a model call — so no
|
||||
source (not even a 1-doc one) is invisible to `<knowledge_base>`
|
||||
(phase 162); the `LLMError`/zero-doc paths leave the previous row
|
||||
(tail included) untouched. Because the sync only rewrites the folders a
|
||||
change touches, a one-shot
|
||||
`uv run python -m scripts.regen_folder_summaries` refreshes EVERY
|
||||
stored folder summary in one run (full mode: every candidate folder
|
||||
@@ -492,11 +497,20 @@ server-side tools — on **grounded** (high-relevance) turns only:
|
||||
|------|-------------|
|
||||
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
|
||||
| **`read(path, offset?, limit?)`** | Appends one indexed document's text to context by its combined `source/path`; optional `offset` (1-based first line) + `limit` (line count) read just that part of a large document; very large reads are truncated at `BOR_READ_MAX_CHARS` with a notice that teaches the workflow — grep reports line numbers, then read that part with `offset` + `limit` |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case- and accent-insensitive fixed substring — accents fold: `besta` matches `BESTÅ`, phase 162); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
|
||||
|
||||
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
|
||||
|
||||
**The prompt teaches the search discipline (phase 162).** A summary or an
|
||||
outline is a hint, never an inventory: the system prompt tells the model to run
|
||||
`ls` with no `path` — the full source list — before ever claiming the knowledge
|
||||
base lacks something, and to `grep` the artifact's part number or a filename
|
||||
token. One grep pattern is one string, so a name's split tokens (`frame`,
|
||||
`white`) and its part number (`AA-1272068`) each need their own grep; and a
|
||||
note saying a file is gitignored describes its **repo** status, not whether it
|
||||
is indexed as a source.
|
||||
|
||||
**Git-source provenance (phase 145).** When a document comes from a git
|
||||
source, its `read` result also carries an `origin:` line (the repository's
|
||||
URL, credentials stripped) and, when the source pins one, a `branch:` line.
|
||||
@@ -583,7 +597,12 @@ stemming) and searched **twice** against Postgres:
|
||||
|
||||
1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`)
|
||||
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
|
||||
top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`)
|
||||
top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`), led by
|
||||
**name hits** — documents whose *path* matches a question token (`gitea`
|
||||
→ `gitea/…`, `qwen38` → `qwen3.8-27b-…`; phase 119). Name normalization
|
||||
is **accent-folded** (phase 162): a question's `BESTÅ` folds to `besta`
|
||||
and matches the ASCII filename, and an accented filename (`BÅsta.md`)
|
||||
matches an ASCII question — both directions, stdlib fold only.
|
||||
|
||||
The two ranked lists are fused with **Reciprocal Rank Fusion**
|
||||
(`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores
|
||||
@@ -917,7 +936,7 @@ to `.env` and adjust.
|
||||
| `BOR_MAX_OUTPUT_TOKENS` | `32768` | Max answer length in tokens (answers must not be cut off) |
|
||||
| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `<tuning>` (steering notes) prompt section |
|
||||
| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model |
|
||||
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `<knowledge_base>` prompt section |
|
||||
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `<knowledge_base>` prompt section — the stored outline **plus** its deterministic `Sources: …` tail (phase 162); the cap keeps the longest-fitting prefix, so a value below the default clips the tail first (the default always holds both) |
|
||||
| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation |
|
||||
| `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` | `8000` | Cap on a folder's document list sent to `lite` for folder summaries (phase 94) |
|
||||
|
||||
@@ -1058,8 +1077,9 @@ normally until you attach. Override the port with `DEBUGPY_PORT`.
|
||||
|
||||
## QA / Testing
|
||||
|
||||
Three layers — the project rule is **one story, one phase, one Playwright
|
||||
suite** (see `AGENTS.md`):
|
||||
Three layers — the project rule is **one phase, one Playwright suite** (a
|
||||
phase may implement a user story or an owner-directed change; see
|
||||
`AGENTS.md`):
|
||||
|
||||
```bash
|
||||
# Unit + integration (FastAPI TestClient)
|
||||
@@ -1075,12 +1095,27 @@ uv run pyright
|
||||
# Playwright E2E — install the browser once:
|
||||
uv run playwright install chromium
|
||||
|
||||
# Each story's E2E runs IN ISOLATION:
|
||||
# Each phase's E2E runs IN ISOLATION:
|
||||
uv run pytest tests/e2e/test_import_documents.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
|
||||
# ...one file per story in .agents/user_stories/ (see .agents/phases/todo/)
|
||||
# ...one file per phase in tests/e2e/ (the stories live in
|
||||
# .agents/user_stories/, see .agents/phases/todo/)
|
||||
```
|
||||
|
||||
**Retrieval regression battery.** `tests/fixtures/retrieval_battery.txt` is
|
||||
the tracked retrieval gate (phases 119 + 162): the name-your-tool questions,
|
||||
the versioned-name (`qwen 3.8 27b`) regression, the out-of-KB deflection
|
||||
sanity, and the `besta` / `BESTÅ` accent-fold pair. Run it against a KB with
|
||||
`scripts.eval_retrieval` (the real pipeline, real embeddings):
|
||||
|
||||
```bash
|
||||
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
|
||||
```
|
||||
|
||||
The exact expected ranks are recorded in the fixture's header comments; a
|
||||
question whose source is missing from the KB snapshot is recorded as skipped
|
||||
loudly, never silently dropped.
|
||||
|
||||
**Deterministic E2E:** by default the E2E app talks to a local **mock
|
||||
aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real token-overlap
|
||||
vectors — so the cosine relevance threshold behaves like production. To
|
||||
|
||||
Reference in New Issue
Block a user