ducoterra
eebc824636
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)** - Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched. - Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated. - Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched). - Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`). - Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382. - `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**. - `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**. - Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness. - Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB. - Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.) - Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass. - Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`". - Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
44 lines
2.3 KiB
Plaintext
44 lines
2.3 KiB
Plaintext
# Retrieval battery — phase 119 (task 06, D6) + phase 162 (task 04, D5),
|
|
# run against the 1,000-doc live replica:
|
|
# uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
|
|
#
|
|
# Gate (per the phase overview, post-phase measured numbers):
|
|
# 1-3. each name question's suggested top-5 contains its canonical doc:
|
|
# gitea -> deploy/Deployments/reeseapps/gitea/README.md (rank 4)
|
|
# forgejo-> 5/5 forgejo-tree docs, README #1
|
|
# gateway-> deploy/Deployments/reeseapps/gateway/README.md (rank 3)
|
|
# 4. the 2026-09-05 incident regression: a qwen3.8-27b quadlet stays in
|
|
# the suggested top-5 (the digit-bearing prefix rule, unchanged).
|
|
# 5. deflection sanity: the out-of-KB question's gate verdict + fts
|
|
# count match the pre-phase replica run (HIGH, fts_hits=5) — the
|
|
# name-hit change must not move it.
|
|
#
|
|
# Phase 162 appends the Besta pair (the 2026-10-07 live miss). Gate: the
|
|
# suggested top-5 must contain the IKEA doc — ASCII filename
|
|
# `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`, body that
|
|
# spells the product `BESTÅ` only — for BOTH spellings: the accented one
|
|
# needs the phase-162 question-side fold (D1) to bridge ASCII filename ←
|
|
# accented question, the ASCII one is the phase-119 path (regression).
|
|
# Either way ASCII `besta` documents must not outrank the doc that names
|
|
# the product.
|
|
#
|
|
# SKIPPED LOUDLY on the local replica (the phase-140 precedent): the
|
|
# 2026-09-05 snapshot carries NO `ikea_instructions`-class source —
|
|
# `grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0, and
|
|
# `SELECT DISTINCT source FROM documents` after the restore lists the
|
|
# sources without it (verify, don't assume). The two questions still
|
|
# run and must behave like any out-of-KB ask (no crash, no false
|
|
# grounding on a doc that never named the product); the gate on the
|
|
# rank itself can only close on a KB that has the source (owner's live
|
|
# KB). The fixture that DOES reproduce the miss end-to-end is
|
|
# tests/e2e/test_search_recall_small_sources.py over
|
|
# tests/fixtures/recallkb (pre-fix ranks measured there, both
|
|
# directions).
|
|
How do I deploy gitea?
|
|
How do I deploy forgejo?
|
|
How do I set up the gateway?
|
|
Do you have notes on the qwen 3.8 27b model?
|
|
What is the capital of Mongolia?
|
|
How do I build a Besta?
|
|
How do I build the BESTÅ?
|