phase: 162_search_recall_small_sources
Build and Push Containers / build-and-push-app (push) Successful in 1m50s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
This commit is contained in:
2026-10-07 01:42:01 -04:00
parent 690fd98036
commit eebc824636
48 changed files with 2685 additions and 80 deletions
@@ -0,0 +1,11 @@
# Pictorial PDF pages
How do I build a retrievable text layer for a scanned manual? Build one page
image per PDF page, build an image description for each page, then build the
index from those descriptions. The ASCII besta assembly PDF was the first
manual to go through this path.
- build page images at 1400 px
- build one description per page image
- build chunks from descriptions, never from pixel guesses
- besta manual: 24 scanned pages, 24 descriptions
@@ -0,0 +1,9 @@
# PDF text backfill
Backfill plan for scanned manuals. A manual that arrives without a text layer
needs one before it can answer anything: run the pictorial page path, build
descriptions for every page, keep the original page order.
- build: render, describe, index
- besta shelf PDF: 24 scanned pages, no embedded text
- question asked twice in review: how do I build this layer?
@@ -0,0 +1,8 @@
# SeaweedFS blob ingest
Object-store notes. Mount first, upload second.
- mount fs://manuals before the first upload
- build the volume list, then copy the scan batch
- besta scan batch: fs://manuals/2026-09/besta-scans.tar
- checksum every archive after the copy
@@ -0,0 +1,8 @@
# History source filter
Turn history keeps the source each answer came from, so a filter can narrow
the log to one source at a time.
- filter chip per source, sourced from turn history
- a build log entry records the source it indexed
- rerunning a question with a filter never rewrites history