phase: 162_search_recall_small_sources
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)** - Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched. - Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated. - Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched). - Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`). - Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382. - `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**. - `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**. - Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness. - Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB. - Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.) - Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass. - Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`". - Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
This commit is contained in:
@@ -307,8 +307,11 @@ def test_mock_generated_outline_is_stored_and_echoed(
|
||||
assert user.splitlines()[0].startswith(
|
||||
"docs — homelab/tables.md — Service Port Table"
|
||||
), "the generator's list must lead with the NEWEST doc (recency-first)"
|
||||
# Phase 162 (LOCKED A2): the stored content ends with the
|
||||
# deterministic source-inventory line — one source, 13 docs.
|
||||
expected = (
|
||||
"Knowledge base outline:\n- docs homelab tables md service port table this"
|
||||
"Knowledge base outline:\n- docs homelab tables md service port table this\n"
|
||||
"Sources: docs (13 docs)"
|
||||
)
|
||||
assert row.content == expected
|
||||
first_bullet = row.content.splitlines()[1].removeprefix("- ").strip()
|
||||
|
||||
@@ -329,8 +329,10 @@ def test_regenerated_outline_leads_with_newest_doc(
|
||||
assert row is not None and row.id == 1
|
||||
# The stored outline is the mock's 8-token digest of the capped list
|
||||
# — all 8 tokens from the NEWEST doc's source/path/title (computed
|
||||
# from the fixture above, not hardcoded wording):
|
||||
assert row.content == EXPECTED_DIGEST
|
||||
# from the fixture above, not hardcoded wording) — and phase 162
|
||||
# (LOCKED A2) appends the deterministic source-inventory line after
|
||||
# it: every fixture source named with its exact count.
|
||||
assert row.content == EXPECTED_DIGEST + "\nSources: reese (5 docs)"
|
||||
assert "old" not in row.content # no oldest-batch token survives the cut
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,864 @@
|
||||
"""Phase 162 E2E (Playwright, mock-only): a 1-document source is BOTH
|
||||
visible to ``<knowledge_base>`` and winnable by its accented question.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov
|
||||
|
||||
The 2026-10-07 live miss, replayed deterministically: the owner asked
|
||||
``How do I build the BESTÅ?``; the KB held ``ikea_instructions/
|
||||
besta-frame-white__AA-1272068-11-100.md`` (1 document in the source, its
|
||||
body carrying the name ONLY as ``BESTÅ``) and the question came back
|
||||
empty-handed — ASCII ``besta`` documents outranked the one document that
|
||||
actually named the product. Three separate losses, three pins here:
|
||||
|
||||
* **D1 (the question-side fold)** — name matching never bridged the
|
||||
accent: ``BESTÅ`` normalized to ``best``, ``besta-frame-…`` to
|
||||
``besta``, and the digitless rule EQUALS a stem component. Test 1
|
||||
pins the fix, test 2 MEASURES the miss.
|
||||
* **D2 (the deterministic source inventory)** — the ``lite`` outline is
|
||||
lossy, so a 1-doc source could be invisible to the model. Test 4 pins
|
||||
the mechanical ``Sources:`` tail on the stored row.
|
||||
* **D3 (the grep fold + search discipline)** — the stored accented body
|
||||
stayed grep-matchable by the ASCII name. Test 6 pins the content
|
||||
layer; the turn-level prose discipline is the prompt teaching
|
||||
(``app/rag/prompts.py``, unit-pinned in task 03).
|
||||
|
||||
KB fixture — ``tests/fixtures/recallkb`` (tracked), THREE local-directory
|
||||
sources registered through the authenticated API + the REAL in-process
|
||||
``POST /api/sync`` (the ``test_name_signal_read_chips.py`` pattern), six
|
||||
markdown documents whose text is token-controlled so the hybrid gate, the
|
||||
FTS list and the name-hit bonus pick the intended tiers deterministically
|
||||
(``synced_kb`` pins the tiers with the app's REAL retrieval — a
|
||||
fixture-text regression breaks at setup with a measured message):
|
||||
|
||||
* ``ikea_instructions/`` — the small source: ONE document, the live
|
||||
incident's own filename, its body naming the product ONLY as the
|
||||
accented ``BESTÅ`` (no ASCII ``besta`` anywhere in the file). Two
|
||||
token controls make the mechanism visible: no ``build`` (the question's
|
||||
other FTS token) anywhere, and the accented name deliberately placed
|
||||
PAST the 24-token digest window, so the mock summary chunk leaks no
|
||||
ASCII ``best`` lexeme that a pre-fix FTS could have hit;
|
||||
* ``phase_docs/`` (4 docs) + ``players_handbook/`` (1 doc) — the ASCII
|
||||
distractors of the live index: every body carries ASCII ``besta`` AND
|
||||
``build``, so each of the five outranks the small source on
|
||||
pre-fix retrieval (two fused terms — FTS + vector — against its one).
|
||||
|
||||
Measured, never assumed (test 2 re-measures both directions; the same
|
||||
numbers are recorded in
|
||||
``.agents/reports/162_search_recall_small_sources/``). Accented question
|
||||
``How do I build the BESTÅ?``, the app's real ``retrieve()`` +
|
||||
``select_suggested()`` over this KB with the mock's embeddings (gate:
|
||||
threshold 0.30, lexical-support floor 0.15 — the conftest mock
|
||||
calibration; ``best cosine`` / ``fts`` are the A8 inputs):
|
||||
|
||||
PRE-FIX (the phase-119 ``_normalize_name``, no accent fold)
|
||||
name tokens ['build', 'best'] → NO name hit
|
||||
suggested the five ASCII distractors
|
||||
related ikea_instructions/besta-frame-white__… (rank 6)
|
||||
best cosine 0.382, fts 8 (grounded, on the WRONG documents —
|
||||
the live 2026-10-07 shape: an answer
|
||||
about the ASCII ``besta`` notes)
|
||||
POST-FIX (shipped)
|
||||
name tokens ['build', 'besta'] → name hit
|
||||
suggested THE SMALL SOURCE #1, then four distractors
|
||||
related phase_docs/156_history_source_filter.md
|
||||
best cosine 0.382, fts 9 (the fold adds the lexical term; the
|
||||
cosine is untouched — LOCKED A1)
|
||||
|
||||
The mechanism is the term COUNT: every distractor matches the question's
|
||||
other FTS token ``build``, so each fuses TWO terms (vector + FTS) while
|
||||
the small source — no ``build``, no accent-bridgeable lexeme — fuses ONE
|
||||
and lands rank 6. The folded name hit is the only signal that can lift
|
||||
it: name hits LEAD the lexical list (``fuse`` rank 1 ⇒ 1/61) and the
|
||||
per-document ``BOR_NAME_HIT_BONUS`` (0.005) rides on top. The ASCII
|
||||
question ``How do I build a Besta?`` keeps the same document suggested
|
||||
#1 with the same cosine, before and after (the untouched phase-119
|
||||
path, test 3).
|
||||
|
||||
Test → contract mapping (task 04 cases 1–4):
|
||||
1. ``test_accented_question_suggests_the_small_source`` — (1): the
|
||||
accented question grounds (``deflected: false``), the durable
|
||||
record's SUGGESTED slice (entries 1–5, LOCKED A3) contains
|
||||
``ikea_instructions/besta-frame-white__AA-1272068-11-100.md``, and
|
||||
the ``done`` frame's ``related`` list — the rank-6+ remainder —
|
||||
does NOT (the doc moved from the rank-6 miss into the tier). A
|
||||
zero-read grounded turn chips nothing (phase 119 LOCKED A1).
|
||||
2. ``test_under_rank_is_measured_not_assumed`` — (1, the hypothesis):
|
||||
with the PRE-FIX normalizer re-installed (``monkeypatch``), the same
|
||||
retrieval over the same KB puts the doc OUT of the top-5 (and the
|
||||
``BESTÅ?`` token degrades to ``best``); with the shipped normalizer
|
||||
it is suggested #1. The under-rank is measured in both directions,
|
||||
so the pin can never silently rot.
|
||||
3. ``test_ascii_question_still_name_matches`` — (2): the ASCII
|
||||
regression (the phase-119 path untouched) — same document in the
|
||||
suggested tier, grounded, same related remainder.
|
||||
4. ``test_overview_tail_names_every_source_with_counts`` — (3): the
|
||||
stored ``kb_overview`` row ENDS with the deterministic
|
||||
``Sources: …`` line, one entry per source with exact counts — the
|
||||
1-doc sources visible whatever the ``lite`` digest said.
|
||||
5. ``test_out_of_kb_question_still_deflects`` — (4): the gate verdict
|
||||
for the out-of-KB question is UNCHANGED (``deflected: true``,
|
||||
``fts_hits == 0``, no chips, ``Maybe try`` chips only) — the fold
|
||||
must not turn a real miss into a false grounding.
|
||||
6. ``test_grep_folds_the_accented_body`` — (3, content layer):
|
||||
``grep_document`` with the ASCII pattern ``besta`` matches the
|
||||
stored accented line (D3), and the accented pattern matches too.
|
||||
7. ``test_suggested_folder_lines_carry_the_small_source`` — the
|
||||
prompt-level witness: the mock's ``repeat your folder map`` echo
|
||||
ends with the suggested-folder lines, and the 1-doc source's line
|
||||
``ikea_instructions/:`` is among them (its suggested doc owns the
|
||||
line, so the header stands alone).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document, KbOverview, QueryLog
|
||||
from app.rag import retriever
|
||||
from app.rag.agent import grep_document
|
||||
from app.rag.retriever import retrieve, select_related, select_suggested
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import TOKEN_RE, embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: Own port (the phase-79 full-inventory rule: a module app binds its
|
||||
#: own port so a combined run never drives the session app).
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_RECALL", "8162"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "recallkb"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The fixture KB (tracked files on disk — read back byte-for-byte)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: source name → document paths. THE SMALL SOURCE is the live incident:
|
||||
#: one document whose body carries the product name only accented.
|
||||
SOURCE_DOCS: dict[str, list[str]] = {
|
||||
"ikea_instructions": ["besta-frame-white__AA-1272068-11-100.md"],
|
||||
"phase_docs": [
|
||||
"140_pictorial_pdf_pages.md",
|
||||
"148_pdf_text_backfill.md",
|
||||
"152_seaweedfs_blob_ingest.md",
|
||||
"156_history_source_filter.md",
|
||||
],
|
||||
"players_handbook": ["game_night_setup.md"],
|
||||
}
|
||||
|
||||
IKEA_SOURCE = "ikea_instructions"
|
||||
IKEA_DOC = "besta-frame-white__AA-1272068-11-100.md"
|
||||
IKEA = f"{IKEA_SOURCE}/{IKEA_DOC}"
|
||||
|
||||
#: Every fixture doc in ``(source, path)`` catalog order.
|
||||
ALL_DOCS = [f"{source}/{path}" for source in SOURCE_DOCS for path in SOURCE_DOCS[source]]
|
||||
|
||||
#: The deterministic source inventory tail (LOCKED A2 order: name-sorted,
|
||||
#: singular ``doc`` at k=1) — the D2 assertion.
|
||||
INVENTORY = "Sources: ikea_instructions (1 doc), phase_docs (4 docs), players_handbook (1 doc)"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The scripted questions
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The live miss, byte-for-byte (the ``Å`` is the whole point — the file
|
||||
#: spells it ``BESTÅ``, the pre-phase name matcher read ``best``).
|
||||
ACCENTED_QUESTION = "How do I build the BESTÅ?"
|
||||
#: The phase-119 regression: the same product, spelled in ASCII. This
|
||||
#: question ALWAYS name-matched (the untouched path).
|
||||
ASCII_QUESTION = "How do I build a Besta?"
|
||||
#: The accented question + the mock's folder-map trigger — the prompt
|
||||
#: lens on the suggested tier (the folder lines are composed from the
|
||||
#: suggested documents only).
|
||||
FOLDER_QUESTION = ACCENTED_QUESTION + " repeat your folder map"
|
||||
#: The out-of-KB control (the phase-119 battery's deflection sanity).
|
||||
DEFLECT_QUESTION = "What is the capital of Mongolia?"
|
||||
|
||||
|
||||
def _expected_summary(content: str, source: str, path: str) -> str:
|
||||
"""The mock lite model's byte-stable digest + the code pointer line
|
||||
(mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch — the
|
||||
first 24 tokens of the document content — plus the summarizer's
|
||||
deterministic ``Source:`` line; no model output is ever trusted)."""
|
||||
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
|
||||
return f"This document covers {digest}.\nSource: {source}/{path}"
|
||||
|
||||
|
||||
def _prefix_normalizer(value: str) -> str:
|
||||
"""The PRE-PHASE-162 name normalizer (the measured before-state):
|
||||
lower + alnum scan with NO accent fold — the exact body
|
||||
``_normalize_name`` had before task 01. Installed by monkeypatch in
|
||||
test 2 so the miss is MEASURED, not asserted from reasoning."""
|
||||
return "".join(retriever._ALNUM_RE.findall(value.lower()))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seed_dirs() -> list[Path]:
|
||||
"""The three local-directory sources (the registered directory's
|
||||
BASENAME is the source name, phase 38). The small source must be a
|
||||
directory of its own — that is what makes it a 1-doc source."""
|
||||
dirs = []
|
||||
for source, paths in SOURCE_DOCS.items():
|
||||
directory = FIXTURES / source
|
||||
assert directory.is_dir(), f"{directory} is missing"
|
||||
for path in paths:
|
||||
assert (directory / path).is_file(), f"fixture doc {source}/{path} is missing"
|
||||
dirs.append(directory)
|
||||
return dirs
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, seed_dirs: list[Path]) -> Iterator[str]:
|
||||
"""The real app under test — per-module app on its own port (the
|
||||
phase-79 inventory pattern), the mock LLM, the mock-calibrated gate,
|
||||
and the code defaults for everything the tiers depend on (suggested
|
||||
5 / related 2 / name-hit bonus 0.005 / recency 0.0007, exactly the
|
||||
production ``.env``-free defaults)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (the conftest pattern): the accented and the
|
||||
# ASCII turns ground on the distractors' cosine (≈0.35); the
|
||||
# out-of-KB control stays under it.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env is the owner's — empty the git list so the registry
|
||||
# holds EXACTLY the three directories this suite registers, and pin
|
||||
# the leak-guarded corpus settings (the conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the tier margins
|
||||
are pinned against EXACTLY these six documents."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the cookie-authenticated status endpoint to a terminal state
|
||||
(the ``test_summary_seed_context`` pattern over plain httpx)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _tiers(question: str) -> tuple[list[str], list[str], float, int]:
|
||||
"""The measured tiers of one question over the synced KB, with the
|
||||
app's REAL hybrid retrieval and the mock's embeddings (the
|
||||
phase-119 measurement lens): ``(suggested, related, top cosine,
|
||||
fts hits)``."""
|
||||
with SessionLocal() as db:
|
||||
chunks = retrieve(db, question, embed_text(question))
|
||||
suggested = select_suggested(chunks)
|
||||
related = select_related(
|
||||
chunks,
|
||||
{d.id for d in suggested},
|
||||
_Settings.model_fields["related_max_docs"].default,
|
||||
)
|
||||
return (
|
||||
[f"{d.source}/{d.path}" for d in suggested],
|
||||
[f"{d.source}/{d.path}" for d in related],
|
||||
max((c.cosine for c in chunks), default=0.0),
|
||||
sum(1 for c in chunks if c.fts_hit),
|
||||
)
|
||||
|
||||
|
||||
def _assert_tiers(
|
||||
question: str, suggested: list[str], related: list[str]
|
||||
) -> None:
|
||||
"""Pin the engineered invariant for one scripted question, with a
|
||||
MEASURED message (a fixture-text regression fails at setup, not at
|
||||
the wire assertions)."""
|
||||
got_suggested, got_related, cosine, fts = _tiers(question)
|
||||
assert got_suggested == suggested, (
|
||||
f"suggested tier drifted for {question!r}: {got_suggested} "
|
||||
f"(expected {suggested}; best cosine {cosine:.3f}, fts {fts})"
|
||||
)
|
||||
assert got_related == related, (
|
||||
f"related tier drifted for {question!r}: {got_related} "
|
||||
f"(expected {related})"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, seed_dirs: list[Path]) -> None:
|
||||
"""The story's precondition: the three-source KB synced under the
|
||||
deterministic mock. Registers the directories through the
|
||||
authenticated API, runs the REAL in-process sync, pins the stored
|
||||
content byte-identical to the tracked fixture files (including the
|
||||
accented ``BESTÅ`` surviving the round trip untouched), pins the
|
||||
fixture's own token controls (the mechanism the miss is reproduced
|
||||
with), and pins the engineered tiers for all three scripted
|
||||
questions."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
for directory in seed_dirs:
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(directory)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == len(ALL_DOCS), body["detail"]
|
||||
assert body["detail"]["pruned"] == 0, body["detail"]
|
||||
|
||||
with SessionLocal() as db:
|
||||
for source, paths in SOURCE_DOCS.items():
|
||||
for path in paths:
|
||||
on_disk = (FIXTURES / source / path).read_text(encoding="utf-8")
|
||||
stored = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source, Document.path == path
|
||||
)
|
||||
)
|
||||
assert stored is not None, f"fixture doc {source}/{path} missing"
|
||||
assert stored.content == on_disk, f"content drifted for {source}/{path}"
|
||||
assert stored.summary == _expected_summary(on_disk, source, path), (
|
||||
f"summary is not the mock digest for {source}/{path}"
|
||||
)
|
||||
schunks = [c for c in stored.chunks if c.is_summary]
|
||||
assert len(schunks) == 1 and schunks[0].embedding is not None, (
|
||||
f"expected one embedded is_summary chunk for {source}/{path}"
|
||||
)
|
||||
|
||||
# ---- The fixture's token controls (the reproduction mechanism) ----
|
||||
ikea_text = (FIXTURES / IKEA_SOURCE / IKEA_DOC).read_text(encoding="utf-8")
|
||||
# (a) the name exists ONLY accented — no ASCII ``besta`` in the body,
|
||||
# so nothing but the phase-162 fold can bridge question ↔ doc;
|
||||
assert "BESTÅ" in ikea_text
|
||||
assert "besta" not in ikea_text.lower().replace("bestå", ""), ikea_text
|
||||
# (b) no ``build`` either: the accented question's only OTHER FTS
|
||||
# token — the small source must win on the NAME signal, never on
|
||||
# a content-word accident;
|
||||
assert "build" not in ikea_text.lower()
|
||||
# (c) the accented name sits PAST the 24-token digest window, so the
|
||||
# mock's ASCII-only digest never leaks a ``best`` lexeme the
|
||||
# pre-phase FTS could have hit (a real ``lite`` summary of the
|
||||
# live PDF never wrote the name in ASCII either).
|
||||
digest = TOKEN_RE.findall(ikea_text.lower())[:24]
|
||||
assert "best" not in digest and "besta" not in digest, digest
|
||||
|
||||
# ---- The engineered tiers (measured, stable across re-imports) ----
|
||||
_assert_tiers(ACCENTED_QUESTION, SUGGESTED_ACCENTED, RELATED_ACCENTED)
|
||||
_assert_tiers(ASCII_QUESTION, SUGGESTED_ASCII, RELATED_ASCII)
|
||||
_assert_tiers(FOLDER_QUESTION, SUGGESTED_FOLDER, RELATED_FOLDER)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped — the
|
||||
scripted turns never change it)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The pinned tiers (MEASURED with the app's real retrieval over the mock
|
||||
# embeddings — see the module docstring's before/after record)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: POST-FIX, accented question: the small source LEADS the tier (its
|
||||
#: name hit leads the lexical list ⇒ ``fuse`` rank 1 = 1/61, plus the
|
||||
#: per-document name-hit bonus 0.005), the five ASCII distractors
|
||||
#: follow in measured retrieval order, one of them falls to rank 6.
|
||||
SUGGESTED_ACCENTED = [
|
||||
IKEA,
|
||||
"phase_docs/140_pictorial_pdf_pages.md",
|
||||
"phase_docs/148_pdf_text_backfill.md",
|
||||
"phase_docs/152_seaweedfs_blob_ingest.md",
|
||||
"players_handbook/game_night_setup.md",
|
||||
]
|
||||
RELATED_ACCENTED = ["phase_docs/156_history_source_filter.md"]
|
||||
#: POST-FIX, ASCII question (the untouched phase-119 path): same doc,
|
||||
#: same #1 slot — the only difference is the measured order of two
|
||||
#: distractors that trade places on the extra ``besta`` FTS term.
|
||||
SUGGESTED_ASCII = [
|
||||
IKEA,
|
||||
"phase_docs/140_pictorial_pdf_pages.md",
|
||||
"phase_docs/148_pdf_text_backfill.md",
|
||||
"players_handbook/game_night_setup.md",
|
||||
"phase_docs/152_seaweedfs_blob_ingest.md",
|
||||
]
|
||||
RELATED_ASCII = ["phase_docs/156_history_source_filter.md"]
|
||||
#: The folder-map turn (the trigger's own tokens dilute the cosine to
|
||||
#: 0.296 — lexical support carries the gate) keeps the same tier.
|
||||
SUGGESTED_FOLDER = list(SUGGESTED_ACCENTED)
|
||||
RELATED_FOLDER = list(RELATED_ACCENTED)
|
||||
#: PRE-FIX (the measured miss, test 2): the folded token never reaches
|
||||
#: ``besta``, so the small source has ONE fused term (vector) against
|
||||
#: every distractor's TWO (vector + FTS ``build``) and the five
|
||||
#: distractors fill the tier — the IKEA doc is rank 6, i.e. the rank-6+
|
||||
#: ``related`` remainder. Exactly the live 2026-10-07 under-rank.
|
||||
PRE_SUGGESTED_ACCENTED = [
|
||||
"phase_docs/140_pictorial_pdf_pages.md",
|
||||
"phase_docs/148_pdf_text_backfill.md",
|
||||
"phase_docs/152_seaweedfs_blob_ingest.md",
|
||||
"players_handbook/game_night_setup.md",
|
||||
"phase_docs/156_history_source_filter.md",
|
||||
]
|
||||
PRE_RELATED_ACCENTED = [IKEA]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the phase-119 house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream (a
|
||||
#: response clone read in the background) — wire-level assertions on the
|
||||
#: ``done`` frame, independent of the rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict[str, Any]]:
|
||||
"""The SSE frames captured since the last submit, once the hook's
|
||||
background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _done(page: Page) -> dict[str, Any]:
|
||||
return next(f for f in _frames(page) if f.get("type") == "done")
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _last_query_log() -> QueryLog:
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(QueryLog)).all()
|
||||
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||
return rows[0]
|
||||
|
||||
|
||||
def _recorded_tiers(row: QueryLog) -> tuple[list[str], list[str]]:
|
||||
"""The durable record's tiers (LOCKED A3: the record is the suggested
|
||||
tier followed by the rank-6+ remainder — the wire's only faithful
|
||||
witness of the suggested set, since ``done.sources`` is the READ-DOCS
|
||||
citation surface and ``done.suggestions`` is the deflection chips)."""
|
||||
recorded = [s for s in row.sources.split(", ") if s]
|
||||
return recorded[:5], recorded[5:]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. (task 04 case 1) the accented question puts the 1-doc source in the
|
||||
# suggested top-5 — and it is no longer the rank-6 miss
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_accented_question_suggests_the_small_source(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""``How do I build the BESTÅ?`` grounds, and the suggested tier of
|
||||
the turn carries ``ikea_instructions/besta-frame-white__AA-1272068-
|
||||
11-100.md`` — SUGGESTED #1, measured. The suggested set is pinned on
|
||||
the wire through the durable record (LOCKED A3: ``sources`` = the
|
||||
suggested 5, then the rank-6+ remainder): the IKEA doc sits inside
|
||||
the first five entries, and the ``done`` frame's ``related`` list —
|
||||
the rank-6+ tier, the slot it occupied as the live miss — is EMPTY
|
||||
of it. A zero-read grounded turn chips nothing (phase 119 LOCKED
|
||||
A1); the answer renders grounded."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, ACCENTED_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = _done(page)
|
||||
assert done["deflected"] is False, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
# The miss was the IKEA doc landing in the rank-6+ remainder. It is
|
||||
# not there anymore — the tier it belongs to is the suggested one.
|
||||
related = [(s["source"], s["path"]) for s in done["related"]]
|
||||
assert (IKEA_SOURCE, IKEA_DOC) not in related, done["related"]
|
||||
# The ``suggestions`` field is the DEFLECTION contract (title chips
|
||||
# under a weak turn) — a grounded turn carries none, so it can never
|
||||
# be the tier witness (the durable record is, and is asserted below).
|
||||
assert done["suggestions"] == [], done["suggestions"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
# The de-emphasized rank-6+ row renders — and the doc it used to
|
||||
# hold (the miss) is a distractor now.
|
||||
row = _last_query_log()
|
||||
related_links = page.locator(".msg.brain .related-docs .related-doc")
|
||||
expect(related_links).to_have_count(len(RELATED_ACCENTED))
|
||||
for i, path in enumerate(RELATED_ACCENTED):
|
||||
expect(related_links.nth(i)).to_contain_text(path)
|
||||
expect(related_links.filter(has_text=IKEA)).to_have_count(0)
|
||||
|
||||
assert row.question == ACCENTED_QUESTION
|
||||
assert row.deflected is False
|
||||
suggested, related_record = _recorded_tiers(row)
|
||||
assert suggested == SUGGESTED_ACCENTED, row.sources
|
||||
assert related_record == RELATED_ACCENTED, row.sources
|
||||
assert suggested[0] == IKEA, row.sources
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. (task 04 case 1, the hypothesis) the under-rank is MEASURED in both
|
||||
# directions: pre-fix the doc loses the tier, post-fix it wins it
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_under_rank_is_measured_not_assumed(
|
||||
monkeypatch: pytest.MonkeyPatch, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The phase-162 hypothesis, measured rather than asserted: with the
|
||||
PRE-phase normalizer re-installed (``lower`` + alnum scan, no accent
|
||||
fold) the accented question degrades to the token ``best`` — no name
|
||||
hit — and the SAME retrieval over the SAME KB drops
|
||||
``ikea_instructions/besta-frame-white__AA-1272068-11-100.md`` out of
|
||||
the suggested top-5 into the rank-6+ remainder (the live miss,
|
||||
reproduced). With the shipped normalizer the folded ``besta`` token
|
||||
name-matches the filename and the doc is suggested #1. Re-running
|
||||
the measurement is the guard: if a future fixture or retriever
|
||||
change ever makes the pre-phase path win, this test fails and the
|
||||
recorded evidence is re-examined instead of trusted."""
|
||||
# The token side of the miss, measured.
|
||||
assert retriever.name_hit_tokens(ACCENTED_QUESTION) == ["build", "besta"]
|
||||
monkeypatch.setattr(retriever, "_normalize_name", _prefix_normalizer)
|
||||
assert retriever.name_hit_tokens(ACCENTED_QUESTION) == ["build", "best"]
|
||||
|
||||
pre_suggested, pre_related, _cosine, _fts = _tiers(ACCENTED_QUESTION)
|
||||
assert IKEA not in pre_suggested, (
|
||||
f"the pre-phase path WINS the tier — the fixture no longer "
|
||||
f"reproduces the miss: {pre_suggested}"
|
||||
)
|
||||
assert pre_suggested == PRE_SUGGESTED_ACCENTED, pre_suggested
|
||||
assert IKEA in pre_related, (
|
||||
f"the miss should surface as the rank-6+ remainder: {pre_related}"
|
||||
)
|
||||
assert pre_related == PRE_RELATED_ACCENTED, pre_related
|
||||
|
||||
monkeypatch.undo()
|
||||
# And the same measurement with the shipped fold: the doc is #1.
|
||||
post_suggested, post_related, _cosine, _fts = _tiers(ACCENTED_QUESTION)
|
||||
assert post_suggested == SUGGESTED_ACCENTED
|
||||
assert post_related == RELATED_ACCENTED
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. (task 04 case 2) the ASCII question keeps its phase-119 name hit
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ascii_question_still_name_matches(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""``How do I build a Besta?`` — the question the phase-119 rule
|
||||
already answered — still grounds on the same document (the fold must
|
||||
be behavior-preserving for the ASCII path: ``besta`` normalizes to
|
||||
``besta`` either way). Same tier position, same durable record,
|
||||
grounded, nothing chipped on a zero-read turn."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, ASCII_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = _done(page)
|
||||
assert done["deflected"] is False, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
expect(_last_brain(page).locator(".bubble")).to_contain_text(
|
||||
MOCK_ANSWER_MARKER, timeout=30_000
|
||||
)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == ASCII_QUESTION
|
||||
assert row.deflected is False
|
||||
suggested, related_record = _recorded_tiers(row)
|
||||
assert suggested == SUGGESTED_ASCII, row.sources
|
||||
assert suggested[0] == IKEA, row.sources
|
||||
assert related_record == RELATED_ASCII, row.sources
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. (task 04 case 3) the deterministic source inventory — the D2 tail
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_overview_tail_names_every_source_with_counts(
|
||||
synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The stored ``kb_overview`` row ENDS with the mechanical
|
||||
``Sources: …`` line — one entry per source, exact counts, singular
|
||||
``doc`` at k=1 — so the 1-document sources are visible to
|
||||
``<knowledge_base>`` whatever the lossy ``lite`` digest said (the
|
||||
2026-10-07 live finding: a 14-source index outlined as 3 groups).
|
||||
The tail is ONE line, appears exactly once, and sits after the
|
||||
model's outline (the model input never sees it)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(KbOverview)).all()
|
||||
assert len(rows) == 1, rows
|
||||
content = rows[0].content
|
||||
lines = content.splitlines()
|
||||
assert lines[-1] == INVENTORY, content
|
||||
assert content.count("Sources: ") == 1, content
|
||||
# The ``lite`` outline still leads (the tail is appended, never a
|
||||
# replacement).
|
||||
assert len(lines) >= 2, content
|
||||
assert lines[0].startswith("Knowledge base outline:"), content
|
||||
for source in SOURCE_DOCS:
|
||||
assert source in lines[-1], content
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 5. (task 04 case 4) the out-of-KB control — the gate verdict unchanged
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_out_of_kb_question_still_deflects(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""``What is the capital of Mongolia?`` matches nothing in this KB:
|
||||
the A8 verdict stays DEFLECTED (``deflected: true``, ``fts_hits ==
|
||||
0``, weak cosine under the mock-calibrated threshold), the answer
|
||||
carries no citation chips, and the only chips are the ``Maybe try``
|
||||
suggestions. The accent fold must never turn a real miss into a
|
||||
false grounding (the phase-119 battery's deflection sanity,
|
||||
localised)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, DEFLECT_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
done = _done(page)
|
||||
assert done["deflected"] is True, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
assert 2 <= len(done["suggestions"]) <= 3, done["suggestions"]
|
||||
|
||||
bubble = _last_brain(page)
|
||||
expect(bubble).to_have_class(re.compile(r"is-deflected"))
|
||||
expect(bubble.locator(".source-chip")).to_have_count(0)
|
||||
expect(
|
||||
bubble.locator(".maybe-try .suggestion-chip")
|
||||
).to_have_count(len(done["suggestions"]))
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == DEFLECT_QUESTION
|
||||
assert row.deflected is True
|
||||
assert row.fts_hits == 0, row.fts_hits
|
||||
assert row.top_score < 0.30, row.top_score
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 6. (task 04 case 3 / D3) the stored accented body stays grep-matchable
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_grep_folds_the_accented_body(synced_kb: None, db_ready: None) -> None:
|
||||
"""The content layer of the same miss: the stored body spells the
|
||||
product ``BESTÅ`` only, and the agent's ``grep`` tool folds BOTH the
|
||||
pattern and the line — so the ASCII pattern ``besta`` finds the
|
||||
accented line (the pre-fold tool returned nothing, which is why the
|
||||
live turn gave up on a document it had already opened), and the
|
||||
accented pattern still matches itself."""
|
||||
with SessionLocal() as db:
|
||||
stored = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == IKEA_SOURCE, Document.path == IKEA_DOC
|
||||
)
|
||||
)
|
||||
assert stored is not None
|
||||
assert "BESTÅ" in stored.content
|
||||
|
||||
ascii_hits = grep_document(stored.content, "besta")
|
||||
assert any("BESTÅ" in line for _no, line in ascii_hits), ascii_hits
|
||||
accented_hits = grep_document(stored.content, "BESTÅ")
|
||||
assert [line for _no, line in accented_hits] == [line for _no, line in ascii_hits]
|
||||
# A pattern that is not the name still finds nothing (the fold is an
|
||||
# accent bridge, not a wildcard).
|
||||
assert grep_document(stored.content, "bestaa") == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 7. the prompt-level witness — the 1-doc source's folder line rides the
|
||||
# HIGH prompt with the suggested tier
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_suggested_folder_lines_carry_the_small_source(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The accented question + the mock's ``repeat your folder map``
|
||||
trigger ⇒ the answer ends with the suggested-folder lines VERBATIM.
|
||||
The 1-doc source contributes ``ikea_instructions/:`` — a header with
|
||||
no sibling entries, because its only document IS the suggested doc —
|
||||
the visible proof the small source reached the prompt with the
|
||||
tier."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, FOLDER_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = _done(page)
|
||||
assert done["deflected"] is False, done
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(bubble).to_contain_text("ikea_instructions/:")
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.deflected is False
|
||||
suggested, _related = _recorded_tiers(row)
|
||||
assert suggested == SUGGESTED_FOLDER, row.sources
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# Assembly Instructions
|
||||
|
||||
Article number 127.206.11 — revision 4
|
||||
|
||||
Cam bolt 130618 · wall bracket 141413 · leveling foot 303154 · dowel 518041
|
||||
|
||||
## Package contents
|
||||
|
||||
- 1 x frame, white, 60 x 20 x 60 cm
|
||||
- 2 x side panels
|
||||
- 1 x back panel, hardboard
|
||||
- 12 x cam bolts with sleeve nuts
|
||||
- 1 bag of dowels
|
||||
|
||||
## Assembly
|
||||
|
||||
Insert sleeve nuts into the drilled holes, draw both side panels against the
|
||||
frame, tighten each cam bolt a quarter turn. Attach back panel with nails
|
||||
supplied, keeping it square and level. Fix wall bracket using plugs matched to
|
||||
wall material — furniture of this depth must always be secured.
|
||||
|
||||
## Product name
|
||||
|
||||
BESTÅ frame, white, 60 x 20 x 60 cm, AA 1272068 11 100. Wipe with a soft
|
||||
cloth dampened in water or a mild cleaner. Never soak the edges.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Pictorial PDF pages
|
||||
|
||||
How do I build a retrievable text layer for a scanned manual? Build one page
|
||||
image per PDF page, build an image description for each page, then build the
|
||||
index from those descriptions. The ASCII besta assembly PDF was the first
|
||||
manual to go through this path.
|
||||
|
||||
- build page images at 1400 px
|
||||
- build one description per page image
|
||||
- build chunks from descriptions, never from pixel guesses
|
||||
- besta manual: 24 scanned pages, 24 descriptions
|
||||
@@ -0,0 +1,9 @@
|
||||
# PDF text backfill
|
||||
|
||||
Backfill plan for scanned manuals. A manual that arrives without a text layer
|
||||
needs one before it can answer anything: run the pictorial page path, build
|
||||
descriptions for every page, keep the original page order.
|
||||
|
||||
- build: render, describe, index
|
||||
- besta shelf PDF: 24 scanned pages, no embedded text
|
||||
- question asked twice in review: how do I build this layer?
|
||||
@@ -0,0 +1,8 @@
|
||||
# SeaweedFS blob ingest
|
||||
|
||||
Object-store notes. Mount first, upload second.
|
||||
|
||||
- mount fs://manuals before the first upload
|
||||
- build the volume list, then copy the scan batch
|
||||
- besta scan batch: fs://manuals/2026-09/besta-scans.tar
|
||||
- checksum every archive after the copy
|
||||
@@ -0,0 +1,8 @@
|
||||
# History source filter
|
||||
|
||||
Turn history keeps the source each answer came from, so a filter can narrow
|
||||
the log to one source at a time.
|
||||
|
||||
- filter chip per source, sourced from turn history
|
||||
- a build log entry records the source it indexed
|
||||
- rerunning a question with a filter never rewrites history
|
||||
@@ -0,0 +1,7 @@
|
||||
# Game night setup
|
||||
|
||||
Room notes, games side of the shelf wall.
|
||||
|
||||
- besta unit holds box library; top shelf for dice trays
|
||||
- build tray insert before next session
|
||||
- keep rules binder beside clock, never stacked on besta top
|
||||
Vendored
+27
-1
@@ -1,4 +1,6 @@
|
||||
# Phase 119 retrieval battery (task 06, D6) — the 1,000-doc live replica.
|
||||
# Retrieval battery — phase 119 (task 06, D6) + phase 162 (task 04, D5),
|
||||
# run against the 1,000-doc live replica:
|
||||
# uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
|
||||
#
|
||||
# Gate (per the phase overview, post-phase measured numbers):
|
||||
# 1-3. each name question's suggested top-5 contains its canonical doc:
|
||||
@@ -10,8 +12,32 @@
|
||||
# 5. deflection sanity: the out-of-KB question's gate verdict + fts
|
||||
# count match the pre-phase replica run (HIGH, fts_hits=5) — the
|
||||
# name-hit change must not move it.
|
||||
#
|
||||
# Phase 162 appends the Besta pair (the 2026-10-07 live miss). Gate: the
|
||||
# suggested top-5 must contain the IKEA doc — ASCII filename
|
||||
# `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`, body that
|
||||
# spells the product `BESTÅ` only — for BOTH spellings: the accented one
|
||||
# needs the phase-162 question-side fold (D1) to bridge ASCII filename ←
|
||||
# accented question, the ASCII one is the phase-119 path (regression).
|
||||
# Either way ASCII `besta` documents must not outrank the doc that names
|
||||
# the product.
|
||||
#
|
||||
# SKIPPED LOUDLY on the local replica (the phase-140 precedent): the
|
||||
# 2026-09-05 snapshot carries NO `ikea_instructions`-class source —
|
||||
# `grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0, and
|
||||
# `SELECT DISTINCT source FROM documents` after the restore lists the
|
||||
# sources without it (verify, don't assume). The two questions still
|
||||
# run and must behave like any out-of-KB ask (no crash, no false
|
||||
# grounding on a doc that never named the product); the gate on the
|
||||
# rank itself can only close on a KB that has the source (owner's live
|
||||
# KB). The fixture that DOES reproduce the miss end-to-end is
|
||||
# tests/e2e/test_search_recall_small_sources.py over
|
||||
# tests/fixtures/recallkb (pre-fix ranks measured there, both
|
||||
# directions).
|
||||
How do I deploy gitea?
|
||||
How do I deploy forgejo?
|
||||
How do I set up the gateway?
|
||||
Do you have notes on the qwen 3.8 27b model?
|
||||
What is the capital of Mongolia?
|
||||
How do I build a Besta?
|
||||
How do I build the BESTÅ?
|
||||
|
||||
@@ -239,10 +239,11 @@ def test_changed_import_writes_overview_row(
|
||||
by_role = {m["role"]: m["content"] for m in llm.chat_calls[3]}
|
||||
assert "FOLDER_SUMMARY_MODE" in by_role["system"]
|
||||
assert by_role["user"].splitlines()[0] == "Folder: MyDocs"
|
||||
# The model's outline lands in the single row.
|
||||
# The model's outline lands in the single row, followed by the
|
||||
# phase-162 deterministic source inventory (LOCKED A2).
|
||||
row = _row(db)
|
||||
assert row is not None
|
||||
assert row.content == "Summary of MyDocs"
|
||||
assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)"
|
||||
assert row.updated_at is not None
|
||||
# The phase's required log line (PLAN §9).
|
||||
assert any("overview: regenerated docs=2 chars=" in r.getMessage() for r in records)
|
||||
@@ -274,7 +275,8 @@ def test_unchanged_reimport_does_not_call_lite(
|
||||
)
|
||||
assert len(llm.chat_calls) == 4 # no new lite call
|
||||
row = _row(db)
|
||||
assert row is not None and row.content == "Summary of MyDocs"
|
||||
assert row is not None
|
||||
assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)"
|
||||
assert _version(db) == 1 # phase 53: an unchanged re-run never bumps
|
||||
|
||||
|
||||
@@ -345,7 +347,8 @@ def test_limit_run_skips_overview(
|
||||
# lite call; the overview + folder gates skip under --limit.
|
||||
assert len(llm.chat_calls) == 5
|
||||
row = _row(db)
|
||||
assert row is not None and row.content == "Summary of MyDocs"
|
||||
assert row is not None
|
||||
assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)"
|
||||
assert _version(db) == 1 # phase 53: --limit debug runs never bump
|
||||
|
||||
|
||||
@@ -462,5 +465,8 @@ def test_overview_outline_names_the_newest_docs_when_the_cap_bites(
|
||||
# would have been "mydocs old 01 md old 01 summary of" (the oldest
|
||||
# batch) — the "summaries only up to phase 113" shape this pin
|
||||
# forbids.
|
||||
assert row.content == "Knowledge base outline:\n- mydocs zz new 03 md zz new 03"
|
||||
assert row.content == (
|
||||
"Knowledge base outline:\n- mydocs zz new 03 md zz new 03\n"
|
||||
"Sources: MyDocs (6 docs)"
|
||||
)
|
||||
assert "old" not in row.content # no oldest-batch token may survive the cut
|
||||
|
||||
@@ -367,7 +367,8 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
# teaching no-match line instead of a trusted miss.
|
||||
assert grep["description"] == (
|
||||
"Search the indexed documents for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines "
|
||||
"(case- and accent-insensitive — accents fold: `besta` "
|
||||
"matches `BESTÅ`) and return up to 20 matching lines "
|
||||
"as `source/path:line: text` — a locator, not a "
|
||||
"context-adder: read the winner with `read` — for a "
|
||||
"large document, use the reported line number as "
|
||||
@@ -387,8 +388,11 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
assert set(grep_params["properties"]) == {"pattern", "path"}
|
||||
assert all(p["type"] == "string" for p in grep_params["properties"].values())
|
||||
assert grep_params["properties"]["pattern"]["description"] == (
|
||||
"The exact text to search for (a plain substring, "
|
||||
"not a regex — no '.*', no '\\.', no character classes)"
|
||||
"The exact text to search for (a plain "
|
||||
"substring — case- and accent-insensitive, "
|
||||
"accents fold: `besta` matches `BESTÅ` — "
|
||||
"not a regex — no '.*', no "
|
||||
"'\\.', no character classes)"
|
||||
)
|
||||
# Phase 72 (task 02): the bare-path contract is stated up front;
|
||||
# task 05 (live gate iterations 1-8): the one-known-document clause
|
||||
@@ -4000,6 +4004,27 @@ def test_origin_lines_image_doc_keeps_marker_position(
|
||||
# ---------- grep (the phase-68 A5 contract under the new name) ----------
|
||||
|
||||
|
||||
def test_grep_document_folds_accents_both_ways() -> None:
|
||||
"""Phase 162 (LOCKED A3): grep folds BOTH the pattern and each line
|
||||
through the shared ``app.rag.retriever._fold`` plus lowercase — an
|
||||
ASCII pattern hits accented content and an accented pattern hits
|
||||
ASCII content, one fold, never drifting from the name-hit path."""
|
||||
# ASCII pattern, accented content.
|
||||
assert agent.grep_document("The BESTÅ frame (AA-1272068-11-100)", "besta") == [
|
||||
(1, "The BESTÅ frame (AA-1272068-11-100)")
|
||||
]
|
||||
assert agent.grep_document("line one\nBÅsta shelf\nline three", "basta") == [
|
||||
(2, "BÅsta shelf")
|
||||
]
|
||||
assert agent.grep_document("café-au-lait", "cafe") == [(1, "café-au-lait")]
|
||||
# Accented pattern, ASCII content — the pattern's own accents fold.
|
||||
assert agent.grep_document("The besta frame", "BESTÅ") == [(1, "The besta frame")]
|
||||
# A genuinely non-matching folded pattern still returns no lines.
|
||||
assert agent.grep_document("The besta frame", "bestb") == []
|
||||
# The fold bridges Ä/Ö/Ü/ß too (the same _fold as name matching).
|
||||
assert agent.grep_document("Straße", "strasse") == [(1, "Straße")]
|
||||
|
||||
|
||||
def test_grep_document_case_insensitive_line_numbers() -> None:
|
||||
"""Case-insensitive fixed substring, 1-based line numbers, file order,
|
||||
repeated matches within a line collapse to one match (grep semantics)."""
|
||||
|
||||
+144
-6
@@ -5,7 +5,11 @@ per-document user lines (source/path/title/first summary line), and the
|
||||
input cap with the shared ``[…truncated…]`` marker. The loader and
|
||||
regenerator tests run against the local compose Postgres (preferred —
|
||||
real single-row upsert), skipping with clear instructions when the stack
|
||||
is not up — same pattern as ``test_importer.py``.
|
||||
is not up — same pattern as ``test_importer.py``. The phase-162 section
|
||||
pins the deterministic ``Sources: …`` inventory tail (LOCKED A2): the
|
||||
format/order/counts, the 1-doc survival, the happy-path stored content,
|
||||
the byte-identical zero-doc / ``LLMError`` paths, and the byte-identical
|
||||
model input.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -15,7 +19,7 @@ import uuid
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
@@ -25,6 +29,7 @@ from app.rag.overview import (
|
||||
KB_OVERVIEW_MODE,
|
||||
OVERVIEW_INSTRUCTION,
|
||||
SYSTEM_PROMPT,
|
||||
_source_inventory,
|
||||
build_overview_prompt,
|
||||
load_kb_overview,
|
||||
regenerate_overview,
|
||||
@@ -302,11 +307,13 @@ def test_regenerate_happy_path_creates_single_row_id_1(
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None, "the single row must be upserted"
|
||||
assert row.id == 1
|
||||
assert row.content == REPLY
|
||||
# Phase 162 (LOCKED A2): the stored content carries the
|
||||
# deterministic source-inventory tail after the model text.
|
||||
assert row.content == REPLY + "\nSources: Homelab (2 docs)"
|
||||
assert row.updated_at is not None
|
||||
age = datetime.now(UTC) - row.updated_at
|
||||
assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp"
|
||||
assert f"overview: regenerated docs=2 chars={len(REPLY)}" in caplog.text
|
||||
assert f"overview: regenerated docs=2 chars={len(row.content)}" in caplog.text
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
@@ -324,7 +331,7 @@ def test_regenerate_updates_existing_row_in_place(db: Session, clean_overview) -
|
||||
assert count == 1, "still exactly one row after the re-generation"
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None
|
||||
assert row.content == REPLY # replaced, not appended
|
||||
assert row.content == REPLY + "\nSources: Homelab (1 doc)" # replaced, not appended
|
||||
assert row.updated_at is not None and row.updated_at > past
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
@@ -511,4 +518,135 @@ def test_regenerate_opens_own_session_when_none(db: Session, clean_overview) ->
|
||||
|
||||
assert ok is True
|
||||
row = db.get(KbOverview, 1) # fresh lookup via the test session
|
||||
assert row is not None and row.content == REPLY
|
||||
assert row is not None and row.content == REPLY + "\nSources: Homelab (1 doc)"
|
||||
|
||||
|
||||
# ---------- phase 162: the deterministic source inventory (LOCKED A2) -------
|
||||
|
||||
|
||||
def test_source_inventory_format_counts_and_catalog_order() -> None:
|
||||
"""Exact format, singular/plural, and catalog order: one entry per
|
||||
DISTINCT source — ``<name> (<k> doc|docs)`` joined with ``, ``
|
||||
behind ``Sources: ``, ordered ``source, path`` (the scrambled
|
||||
input rows prove the catalog order is the OUTPUT order, not the
|
||||
recency-first order of the rows the generator reads)."""
|
||||
rows = [
|
||||
("b", "x.md", "X", None),
|
||||
("a", "y.md", "Y", None),
|
||||
("c", "p1.md", "P1", None),
|
||||
("b", "z.md", "Z", None),
|
||||
("c", "p2.md", "P2", None),
|
||||
("c", "p3.md", "P3", None),
|
||||
]
|
||||
assert _source_inventory(rows) == "Sources: a (1 doc), b (2 docs), c (3 docs)"
|
||||
|
||||
|
||||
def test_source_inventory_keeps_a_one_doc_source_visible() -> None:
|
||||
"""The 2026-10-07 ``ikea_instructions`` case: a 1-doc source among
|
||||
many appears with ``(1 doc)`` — no source is invisible to
|
||||
``<knowledge_base>``, whatever the ``lite`` outline says."""
|
||||
rows = [
|
||||
("gufo", "a.md", "A", None),
|
||||
(
|
||||
"ikea_instructions",
|
||||
"besta-frame-white__AA-1272068-11-100.md",
|
||||
"Assembly Instructions",
|
||||
None,
|
||||
),
|
||||
("gufo", "b.md", "B", None),
|
||||
("players_handbook", "core.md", "Core", None),
|
||||
]
|
||||
assert _source_inventory(rows) == (
|
||||
"Sources: gufo (2 docs), ikea_instructions (1 doc), "
|
||||
"players_handbook (1 doc)"
|
||||
)
|
||||
|
||||
|
||||
def test_regenerate_appends_the_inventory_tail_to_the_stored_content(
|
||||
db: Session, clean_overview, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Happy path, exact pin on the fake's fixed output: the stored row
|
||||
= model text (trailing newlines rstripped) + ``\\n`` + the inventory
|
||||
line; the log's chars = the FULL stored content, tail included."""
|
||||
_truncate_documents(db)
|
||||
try:
|
||||
_add_doc(db, "players_handbook", "rpg.md", "RPG")
|
||||
_add_doc(db, "Homelab", "b.md", "B")
|
||||
_add_doc(db, "Homelab", "a.md", "A")
|
||||
llm = _FakeLLM(reply=REPLY + "\n\n")
|
||||
expected = REPLY + "\nSources: Homelab (2 docs), players_handbook (1 doc)"
|
||||
with caplog.at_level(logging.INFO, logger="app.rag.overview"):
|
||||
ok = asyncio.run(regenerate_overview(llm, db))
|
||||
assert ok is True
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None
|
||||
assert row.content == expected # no blank line before the tail
|
||||
assert f"overview: regenerated docs=3 chars={len(expected)}" in caplog.text
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
|
||||
def test_regenerate_model_input_never_gains_the_inventory_tail(
|
||||
db: Session, clean_overview
|
||||
) -> None:
|
||||
"""The model input is byte-identical to the pre-phase prompt: the
|
||||
user message the fake received equals ``build_overview_prompt`` for
|
||||
the same catalogue rows — the tail is never sent to ``lite``."""
|
||||
_truncate_documents(db)
|
||||
try:
|
||||
_add_doc(db, "Homelab", "a.md", "A", "A summary.\nSource: Homelab/a.md")
|
||||
_add_doc(db, "Deployments", "b.yaml", "B")
|
||||
llm = _FakeLLM()
|
||||
assert asyncio.run(regenerate_overview(llm, db)) is True
|
||||
rows = db.execute(
|
||||
select(Document.source, Document.path, Document.title, Document.summary)
|
||||
.order_by(Document.created_at.desc(), Document.source, Document.path)
|
||||
).all()
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
user = next(m["content"] for m in llm.messages if m["role"] == "user")
|
||||
_, expected_user = build_overview_prompt([tuple(r) for r in rows])
|
||||
assert user == expected_user
|
||||
assert "Sources:" not in user
|
||||
|
||||
|
||||
def test_regenerate_zero_docs_keeps_tailless_pre_phase_row(
|
||||
db: Session, clean_overview
|
||||
) -> None:
|
||||
"""Zero-doc early return: byte-identical pre-phase behavior — the
|
||||
pre-phase (tail-less) row survives untouched, no tail is invented."""
|
||||
db.execute(text("DELETE FROM kb_overview"))
|
||||
pre_phase_row = "- Old outline\n- from before phase 162"
|
||||
db.add(KbOverview(id=1, content=pre_phase_row))
|
||||
db.commit()
|
||||
_truncate_documents(db)
|
||||
llm = _FakeLLM()
|
||||
ok = asyncio.run(regenerate_overview(llm, db))
|
||||
assert ok is False
|
||||
assert llm.calls == 0
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None and row.content == pre_phase_row
|
||||
assert "Sources:" not in row.content
|
||||
|
||||
|
||||
def test_regenerate_llm_error_preserves_an_existing_inventory_tail(
|
||||
db: Session, clean_overview
|
||||
) -> None:
|
||||
"""The tail is CONTENT: a previous row that already carried a
|
||||
``Sources: …`` tail survives an ``LLMError`` byte-identically —
|
||||
including the tail (an old outline is better than none)."""
|
||||
db.execute(text("DELETE FROM kb_overview"))
|
||||
previous = "- Old outline\nSources: Homelab (3 docs)"
|
||||
db.add(KbOverview(id=1, content=previous))
|
||||
db.commit()
|
||||
_truncate_documents(db)
|
||||
try:
|
||||
_add_doc(db, "Homelab", "a.md", "A")
|
||||
llm = _FakeLLM(fail=LLMError("simulated lite-model failure"))
|
||||
ok = asyncio.run(regenerate_overview(llm, db))
|
||||
assert ok is False
|
||||
row = db.get(KbOverview, 1)
|
||||
assert row is not None and row.content == previous
|
||||
finally:
|
||||
_truncate_documents(db)
|
||||
|
||||
@@ -17,6 +17,7 @@ import hashlib
|
||||
from app.rag.prompts import (
|
||||
ORIGIN_RULE,
|
||||
PERSONA,
|
||||
SEARCH_DISCIPLINE_RULE,
|
||||
SUGGEST_INTRO,
|
||||
TOOLS_SECTION,
|
||||
_base,
|
||||
@@ -87,9 +88,34 @@ def test_high_and_low_bases_byte_locked() -> None:
|
||||
#: inserted — the prefix (the ``ls``-clause opening) and the suffix
|
||||
#: (the discipline-rules ending) survived byte-identical, so they are
|
||||
#: the same anchors as pre-phase-145; the pre-phase-145 sha/length are
|
||||
#: kept below for the diff-style pin.
|
||||
TOOLS_SECTION_SHA256 = "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
|
||||
TOOLS_SECTION_LEN = 3279
|
||||
#: kept below for the chained diff-style pin. RE-CUT again for phase 162
|
||||
#: (task 03, LOCKED A3+A4): the ``grep`` sentence's ``(case-insensitive)``
|
||||
#: parenthetical widened to the folded contract ``(case- and
|
||||
#: accent-insensitive — accents fold: besta matches BESTÅ)``, and the
|
||||
#: LOCKED search-discipline sentence (:data:`SEARCH_DISCIPLINE_RULE`) was
|
||||
#: inserted immediately after the grep sentence (before the ``For a normal
|
||||
#: search pass`` clause). Only those two byte-groups moved — the prefix
|
||||
#: (the ``ls``-clause opening) and the suffix (the discipline-rules
|
||||
#: ending) survive byte-identical; the pre-phase-162 sha/length are kept
|
||||
#: below for the chained diff-style pin.
|
||||
TOOLS_SECTION_SHA256 = "988642f9dc6bb2a4a1d44a83b5368ff4cdb9df4e181b2fe3dc2fb241b1857595"
|
||||
TOOLS_SECTION_LEN = 3818
|
||||
|
||||
# The pre-phase-162 anchors — the section BEFORE phase 162's grep-fold +
|
||||
# search-discipline insertion. Kept for the chained diff-style pin.
|
||||
TOOLS_SECTION_PRE_PHASE_162_SHA256 = (
|
||||
"41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
|
||||
)
|
||||
TOOLS_SECTION_PRE_PHASE_162_LEN = 3279
|
||||
|
||||
# The grep contract parenthetical exactly as it changed in phase 162
|
||||
# (LOCKED A3): the pre-phase ``(case-insensitive)`` vs the folded one now
|
||||
# embedded in ``TOOLS_SECTION``. Used to revert phase 162 for the diff pin.
|
||||
_GREP_CONTRACT_PRE_PHASE_162 = "(case-insensitive)"
|
||||
_GREP_CONTRACT_PHASE_162 = (
|
||||
"(case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)"
|
||||
)
|
||||
|
||||
TOOLS_SECTION_PREFIX = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
@@ -159,19 +185,46 @@ def test_tools_section_origin_rule_verbatim_at_its_position() -> None:
|
||||
) in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_origin_rule_is_the_only_byte_change() -> None:
|
||||
"""Phase 145 (task 02) diff-style pin: the section's OTHER bytes are
|
||||
byte-identical to pre-phase — the pre-phase section with ONLY the new
|
||||
sentence-group inserted. Removing the rule (and the single space
|
||||
that now separates it from the ``grep`` clause) reproduces the
|
||||
pre-phase-145 section byte-for-byte (sha + length), and the length
|
||||
delta is exactly the rule's length plus that one inserted space."""
|
||||
assert len(TOOLS_SECTION) == (
|
||||
TOOLS_SECTION_PRE_PHASE_145_LEN + len(ORIGIN_RULE) + 1
|
||||
)
|
||||
pre_phase = TOOLS_SECTION.replace(ORIGIN_RULE + " ", "")
|
||||
assert len(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_LEN
|
||||
assert _sha256(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_SHA256
|
||||
def test_tools_section_search_discipline_rule_verbatim_at_its_position() -> None:
|
||||
"""Phase 162 (task 03, LOCKED A4): the search-discipline rule is
|
||||
LOCKED verbatim and sits at its EXACT position — immediately after
|
||||
the grep sentence (the ``…retry with the plain text you expect to
|
||||
see.`` clause) and BEFORE the ``For a normal search pass`` clause.
|
||||
The rule + its exact boundaries are pinned as one contiguous span, so
|
||||
a move or re-spelling of ANY byte fails."""
|
||||
assert SEARCH_DISCIPLINE_RULE in TOOLS_SECTION
|
||||
assert TOOLS_SECTION.count(SEARCH_DISCIPLINE_RULE) == 1 # exactly once
|
||||
assert (
|
||||
"retry with the plain text you expect to see. "
|
||||
+ SEARCH_DISCIPLINE_RULE
|
||||
+ " For a normal search pass only `pattern`"
|
||||
) in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_grep_fold_contract_present() -> None:
|
||||
"""Phase 162 (task 03, LOCKED A3): the grep sentence now states the
|
||||
FOLDED contract — the ``(case-insensitive)`` parenthetical is replaced
|
||||
byte-for-byte by the folded one, and the old one is gone."""
|
||||
assert _GREP_CONTRACT_PHASE_162 in TOOLS_SECTION
|
||||
assert _GREP_CONTRACT_PRE_PHASE_162 not in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_phase162_and_145_are_the_only_byte_changes() -> None:
|
||||
"""Chained diff-style pin: the ONLY byte changes are phase 162's (the
|
||||
folded grep contract + :data:`SEARCH_DISCIPLINE_RULE`) and phase 145's
|
||||
(:data:`ORIGIN_RULE`). Reverting phase 162 (drop the rule + its
|
||||
separating space, restore the pre-phase grep parenthetical) reproduces
|
||||
the pre-phase-162 section byte-for-byte; reverting the origin rule on
|
||||
top of that reproduces the pre-phase-145 section byte-for-byte."""
|
||||
# Revert phase 162 -> pre-phase-162.
|
||||
pre_162 = TOOLS_SECTION.replace(SEARCH_DISCIPLINE_RULE + " ", "")
|
||||
pre_162 = pre_162.replace(_GREP_CONTRACT_PHASE_162, _GREP_CONTRACT_PRE_PHASE_162)
|
||||
assert len(pre_162) == TOOLS_SECTION_PRE_PHASE_162_LEN
|
||||
assert _sha256(pre_162) == TOOLS_SECTION_PRE_PHASE_162_SHA256
|
||||
# Revert phase 145 -> pre-phase-145.
|
||||
pre_145 = pre_162.replace(ORIGIN_RULE + " ", "")
|
||||
assert len(pre_145) == TOOLS_SECTION_PRE_PHASE_145_LEN
|
||||
assert _sha256(pre_145) == TOOLS_SECTION_PRE_PHASE_145_SHA256
|
||||
|
||||
|
||||
# ---------- the LOW prompt's locked DEFLECT_MODE body ----------
|
||||
|
||||
+19
-10
@@ -50,6 +50,7 @@ from app.models import Document
|
||||
from app.rag.prompts import (
|
||||
ORIGIN_RULE,
|
||||
PERSONA,
|
||||
SEARCH_DISCIPLINE_RULE,
|
||||
SHOW_IMAGE_CLAUSE,
|
||||
SUGGEST_INTRO,
|
||||
TOOLS_SECTION,
|
||||
@@ -286,14 +287,14 @@ def test_tools_section_markers_and_new_tool_names() -> None:
|
||||
|
||||
def test_tools_section_teaches_the_harness_shapes() -> None:
|
||||
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
|
||||
case-insensitive exact-string locator contract (up to 20
|
||||
case- and accent-insensitive exact-string locator contract (up to 20
|
||||
``source/path:line: text`` lines, a locator not a context-adder),
|
||||
and ``read``'s combined ``source/path`` + full content. (Phase 72:
|
||||
the source-name scope clause and the combined-identity clause are
|
||||
pinned byte-for-byte in
|
||||
:func:`test_tools_section_phase72_contract_clauses`.)"""
|
||||
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
|
||||
assert "case-insensitive" in TOOLS_SECTION
|
||||
assert "case- and accent-insensitive" in TOOLS_SECTION # phase 162 fold
|
||||
assert "up to 20" in TOOLS_SECTION
|
||||
assert "source/path:line: text" in TOOLS_SECTION
|
||||
assert "locator, not a context-adder" in TOOLS_SECTION
|
||||
@@ -414,20 +415,28 @@ def test_tools_section_phase118_summary_seed_read_clause() -> None:
|
||||
"| path: Y | title: Z`; to find one specific document without "
|
||||
"listing, use `grep`."
|
||||
) in TOOLS_SECTION
|
||||
# The grep clause — byte-identical.
|
||||
# The grep clause — folded contract (phase 162, LOCKED A3), up to the
|
||||
# search-discipline sentence, byte-identical.
|
||||
assert (
|
||||
"`grep` locates an exact string (case-insensitive) in the "
|
||||
"`grep` locates an exact string (case- and accent-insensitive — "
|
||||
"accents fold: `besta` matches `BESTÅ`) in the "
|
||||
"indexed documents and returns up to 20 matching "
|
||||
"`source/path:line: text` lines — a locator, not a "
|
||||
"context-adder: read the winner with `read`. A grep pattern "
|
||||
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
|
||||
"literal text there; if such a pattern returns no matches, "
|
||||
"retry with the plain text you expect to see. For a normal "
|
||||
"search pass only `pattern` — its optional `path` argument "
|
||||
"limits the search to one document you already know, by the "
|
||||
"same combined `source/path` string; never a source name — a "
|
||||
"bare document path (without the source name) will not "
|
||||
"resolve there either."
|
||||
"retry with the plain text you expect to see. "
|
||||
) in TOOLS_SECTION
|
||||
# The LOCKED search-discipline sentence sits immediately after the
|
||||
# grep sentence, before the path clause (phase 162, LOCKED A4).
|
||||
assert (
|
||||
"retry with the plain text you expect to see. "
|
||||
+ SEARCH_DISCIPLINE_RULE
|
||||
+ " For a normal search pass only `pattern` — its optional "
|
||||
"`path` argument limits the search to one document you already "
|
||||
"know, by the same combined `source/path` string; never a "
|
||||
"source name — a bare document path (without the source name) "
|
||||
"will not resolve there either."
|
||||
) in TOOLS_SECTION
|
||||
# The discipline rules — byte-identical.
|
||||
assert (
|
||||
|
||||
@@ -1346,3 +1346,156 @@ def test_bonus_lives_in_the_selection_layer_only() -> None:
|
||||
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
|
||||
}
|
||||
assert before == after
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 162, D1 — the accent-folded name normalization (LOCKED A1): the
|
||||
# 2026-10-07 live gap, the ASCII filename vs the accented content layer
|
||||
# (``besta`` <-> ``BESTÅ``, both directions, ONE function)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from app.rag.retriever import _fold, _name_parts # noqa: E402
|
||||
|
||||
|
||||
def test_fold_accent_ligature_and_sharp_s_cases() -> None:
|
||||
"""The shared accent fold (phase 162, LOCKED A1): NFKD decomposition,
|
||||
combining marks dropped — ``Å/å → a``, ``Ä/Ö/Ü → a/o/u`` (the letter
|
||||
survives, the mark drops), ``ß → ss`` (the Unicode fold —
|
||||
``str.casefold``; NFKD alone leaves ``ß`` intact), the ``fi`` ligature
|
||||
``→ fi``, ``ç → c``."""
|
||||
assert _fold("BESTÅ") == "besta"
|
||||
assert _fold("Åsa") == "asa"
|
||||
assert _fold("ÄÖÜäöü") == "aouaou"
|
||||
assert _fold("Straße") == "strasse"
|
||||
assert _fold("fine") == "fine"
|
||||
assert _fold("garçon") == "garcon"
|
||||
|
||||
|
||||
def test_fold_survives_non_decomposing_characters() -> None:
|
||||
"""Characters with NO decomposition (CJK) survive the fold unchanged —
|
||||
the caller's alnum scan drops them, exactly as pre-phase. Compat-only
|
||||
characters (``①``) NFKD to their ASCII base (``1``); that base is
|
||||
below :data:`NAME_TOKEN_MIN_LEN`, so it never becomes a name token
|
||||
(the name-signal behavior for such characters is unchanged)."""
|
||||
assert _fold("漢字") == "漢字"
|
||||
assert _fold("①") == "1"
|
||||
|
||||
|
||||
def test_normalize_name_is_accent_folded() -> None:
|
||||
"""``_normalize_name`` = the alnum scan of ``_fold(s.lower())`` —
|
||||
``Qwen 3.8 → qwen38`` UNCHANGED, ``BESTÅ → besta`` NEW (pre-phase it
|
||||
was ``best``); non-folding characters still drop out (``漢字 → ""``)."""
|
||||
assert _normalize_name("Qwen 3.8") == "qwen38"
|
||||
assert _normalize_name("BESTÅ") == "besta"
|
||||
assert _normalize_name("BÅsta") == "basta"
|
||||
assert _normalize_name("straße") == "strasse"
|
||||
assert _normalize_name("fine") == "fine"
|
||||
assert _normalize_name("garçon") == "garcon"
|
||||
assert _normalize_name("漢字") == ""
|
||||
# Compat-only base survives (``①`` → ``1``) but is a 1-char token —
|
||||
# below the minimum, never a name candidate (see the token test).
|
||||
assert _normalize_name("①") == "1"
|
||||
|
||||
|
||||
def test_name_hit_tokens_accented_question_bridges() -> None:
|
||||
"""The question side of the bridge: the 2026-10-07 battery question
|
||||
produces the ``besta`` token (pre-phase: ``best`` — the exact list is
|
||||
pinned; ``how``/``do``/``i``/``the`` are below NAME_TOKEN_MIN_LEN).
|
||||
The fold covers every pinned case: ``BESTÅ``, umlauts, ``ß``, the
|
||||
``fi`` ligature, ``ç``."""
|
||||
assert name_hit_tokens("How do I build the BESTÅ?") == ["build", "besta"]
|
||||
assert name_hit_tokens("how do I fine the Über strasse") == [
|
||||
"fine", "uber", "strasse",
|
||||
]
|
||||
assert name_hit_tokens("MÖRK ÖVER BÄSTA GARÇON") == [
|
||||
"mork", "over", "basta", "garcon",
|
||||
]
|
||||
|
||||
|
||||
def test_name_hit_tokens_min_length_on_the_folded_form() -> None:
|
||||
"""NAME_TOKEN_MIN_LEN is measured on the FOLDED form (``BÄST`` →
|
||||
``bast`` (4, kept) while ``BÄ`` → ``ba`` (2, dropped)), and
|
||||
non-folding tokens still drop out: ``漢字`` normalizes to ``""`` and
|
||||
``①`` to the 1-char ``1`` — neither can ever be a candidate. The
|
||||
``1`` is compat-decomposition (NFKD — LOCKED A1): the ``①`` token
|
||||
itself never survives, and the only footprint a digit-base can leave
|
||||
is a versioned-JOIN artifact (``the`` + ``1`` → ``the1``) whose sole
|
||||
match route is PREFIXING a normalized path part — no real path does,
|
||||
so the name signal for such characters is unchanged."""
|
||||
assert name_hit_tokens("BÄST BÄ") == ["bast"]
|
||||
assert name_hit_tokens("tell me about it ① 漢字") == ["tell", "about"]
|
||||
# The compat-decomposed ``1`` acts as a version token for the join —
|
||||
# pinned as the real behavior (a prefix-class candidate no path hits).
|
||||
assert name_hit_tokens("tell me about the ① 漢字") == [
|
||||
"tell", "about", "the1",
|
||||
]
|
||||
|
||||
|
||||
def test_name_parts_fold_both_the_ascii_and_accented_stem() -> None:
|
||||
"""The path side of the bridge: the ASCII IKEA filename yields
|
||||
``besta`` in the EQUAL set (the stem sub-component — the phase-119
|
||||
A2 case, unchanged), and the accented filename ``BÅsta.md`` yields
|
||||
``basta`` (the folded stem — already in the equal set via the
|
||||
normalized-stem member, no _name_parts code change)."""
|
||||
_prefix, equal = _name_parts(
|
||||
"ikea/besta-frame-white__AA-1272068-11-100.pdf"
|
||||
)
|
||||
assert "besta" in equal # sub-component, unchanged
|
||||
_prefix2, equal2 = _name_parts("manuals/BÅsta.md")
|
||||
assert "basta" in equal2 # folded stem — an ASCII question matches it
|
||||
|
||||
|
||||
def test_name_hit_chunks_accented_question_hits_ascii_doc() -> None:
|
||||
"""The bridge, end to end (direction 1): the accented question
|
||||
``BESTÅ`` name-hits the ASCII-stem document — the token ``besta``
|
||||
EQUALS the stem sub-component of
|
||||
``besta-frame-white__AA-1272068-11-100`` (the 2026-10-07 live miss;
|
||||
pre-phase the token was ``best`` and the hit was impossible)."""
|
||||
ikea = _doc(
|
||||
"ikea/besta-frame-white__AA-1272068-11-100.pdf",
|
||||
"MONTERING BÄSTA",
|
||||
source="ikea_instructions",
|
||||
)
|
||||
distractor = _doc("notes/meeting.md", "we discussed besta", source="Brain-of-Reese")
|
||||
session = _FakeSession(
|
||||
[_name_row(ikea), _name_row(distractor)],
|
||||
[_name_hit_lateral_row(ikea)],
|
||||
)
|
||||
out = _name_hit_chunks(session, "How do I build the BESTÅ?") # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [ikea.path]
|
||||
assert out[0].name_hit is True
|
||||
assert out[0].fts_hit is True
|
||||
assert out[0].cosine == 0.0
|
||||
|
||||
|
||||
def test_name_hit_chunks_ascii_question_hits_accented_doc() -> None:
|
||||
"""The bridge, end to end (direction 2): the ASCII question token
|
||||
``basta`` name-hits the ACCENTED filename ``BÅsta.md`` — the folded
|
||||
stem enters the equal set as ``basta`` (pre-phase the accented stem
|
||||
normalized to ``bsta`` — the ``å`` simply dropped out of the alnum
|
||||
scan — so the match was impossible)."""
|
||||
accented = _doc("manuals/BÅsta.md", "MONTERING", source="ikea_instructions")
|
||||
miss = _doc("manuals/hesta.md", "MONTERING", source="ikea_instructions")
|
||||
session = _FakeSession(
|
||||
[_name_row(accented), _name_row(miss)],
|
||||
[_name_hit_lateral_row(accented)],
|
||||
)
|
||||
out = _name_hit_chunks(session, "How do I assemble the basta shelf?") # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [accented.path]
|
||||
assert out[0].name_hit is True
|
||||
|
||||
|
||||
def test_name_hit_chunks_fold_preserves_digitless_precision() -> None:
|
||||
"""The fold does NOT relax the A2 match classes: the accented token
|
||||
``déploy`` folds to ``deploy`` and STILL does not hit the
|
||||
``Deployments/`` document (the part normalizes to ``deployments`` ≠
|
||||
``deploy`` — titles are never matched, digits never enter); only the
|
||||
exact ``gitea`` component hits."""
|
||||
precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
|
||||
gitea = _doc("notes/gitea.md", "body")
|
||||
session = _FakeSession(
|
||||
[_name_row(precision), _name_row(gitea)],
|
||||
[_name_hit_lateral_row(gitea)],
|
||||
)
|
||||
out = _name_hit_chunks(session, "how do I déploy gitea") # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [gitea.path]
|
||||
|
||||
Reference in New Issue
Block a user