diff --git a/.agents/phases/todo/119_name_signal_read_chips/00_phase.md b/.agents/phases/complete/119_name_signal_read_chips/00_phase.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/00_phase.md rename to .agents/phases/complete/119_name_signal_read_chips/00_phase.md diff --git a/.agents/phases/todo/119_name_signal_read_chips/01_name_hit_component_match.md b/.agents/phases/complete/119_name_signal_read_chips/01_name_hit_component_match.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/01_name_hit_component_match.md rename to .agents/phases/complete/119_name_signal_read_chips/01_name_hit_component_match.md diff --git a/.agents/phases/todo/119_name_signal_read_chips/02_name_hit_bonus.md b/.agents/phases/complete/119_name_signal_read_chips/02_name_hit_bonus.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/02_name_hit_bonus.md rename to .agents/phases/complete/119_name_signal_read_chips/02_name_hit_bonus.md diff --git a/.agents/phases/todo/119_name_signal_read_chips/03_suggested_folder_context.md b/.agents/phases/complete/119_name_signal_read_chips/03_suggested_folder_context.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/03_suggested_folder_context.md rename to .agents/phases/complete/119_name_signal_read_chips/03_suggested_folder_context.md diff --git a/.agents/phases/todo/119_name_signal_read_chips/04_cite_discipline_copy.md b/.agents/phases/complete/119_name_signal_read_chips/04_cite_discipline_copy.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/04_cite_discipline_copy.md rename to .agents/phases/complete/119_name_signal_read_chips/04_cite_discipline_copy.md diff --git a/.agents/phases/todo/119_name_signal_read_chips/05_chips_read_only.md b/.agents/phases/complete/119_name_signal_read_chips/05_chips_read_only.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/05_chips_read_only.md rename to .agents/phases/complete/119_name_signal_read_chips/05_chips_read_only.md diff --git a/.agents/phases/todo/119_name_signal_read_chips/06_e2e_retarget_and_battery.md b/.agents/phases/complete/119_name_signal_read_chips/06_e2e_retarget_and_battery.md similarity index 100% rename from .agents/phases/todo/119_name_signal_read_chips/06_e2e_retarget_and_battery.md rename to .agents/phases/complete/119_name_signal_read_chips/06_e2e_retarget_and_battery.md diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.md new file mode 100644 index 0000000..4d84026 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.md @@ -0,0 +1,10 @@ +All verification complete. Final report: + +**Phase 119 final verification pass — all criteria verified, one stale pin fixed.** +- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. +- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. +- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. +- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. +- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). +- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). +- Next pending phase: **none** — `todo/` holds only phase 119. diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.validate new file mode 100644 index 0000000..c6535d5 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__00_phase.a1.validate @@ -0,0 +1,104 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 39%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 81%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 96%] +........................................................................ [ 98%] +........................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 226 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 231 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 347 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 102 0 100% +app/rag/retriever.py 242 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4269 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.md new file mode 100644 index 0000000..31d105f --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.md @@ -0,0 +1,19 @@ +**Task 01 complete — component name-hit rule (phase-119 D1, LOCKED A2)** + +Implemented: +- `name_hit_tokens`: class-agnostic candidates (all normalized tokens ≥4, dotted whole + versioned join); digit filter moved to the match side +- `_name_hit_chunks`: two-class rule via new `_name_parts`/`_name_token_matches` — digit-bearing → prefix of normalized part/stem; digitless → equals part/stem/stem sub-component; titles never matched; ranking (matched-token count desc, catalog order) — length tie-break retired; cap 10 and LATERAL representative fetch unchanged +- `RetrievedChunk.name_hit: bool = False` (after `is_summary`); `fuse()` double-hit merge ORs the flag in so the flag survives fusion +- Module/function docstrings re-cut for the two-class rule + title-exclusion guard + +Test/lint/coverage (all green): +- `uv run pytest tests/unit/test_retriever.py -v` → 64 passed (new pins: digitless part/stem/sub-component hits, "deploy"/"Deployments" precision, `qwen38` prefix vs mid-word miss, short tokens, count-then-catalog ranking with the old-order flip, cap, flag-through-fuse) +- `uv run pytest --cov=app --cov-report=term` → 2514 passed, app/ TOTAL 99% (>90%) +- `uv run ruff check . && uv run pyright` → clean, 0 errors + +Decisions: +- `fuse()` OR-propagation (documented in docstring) so task-02's bonus input is visible on merged double-hit rows +- Phase-106 recency owner-scenario re-pinned: the `retention*` docs are now legitimate name hits, margin 0.00262816 → exactly 1/63−1/70 = 0.00158730; flip-safety bar (margin > zero-age boost) kept at 2.27×, A still first +- Integration name-hit suite updated: `llama.cpp.md` now also name-hits (question names it), limit-test stems renamed to prefix form + +Next pending task: `.agents/phases/todo/119_name_signal_read_chips/02_name_hit_bonus.md` diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.validate new file mode 100644 index 0000000..ce1d0c7 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__01_name_hit_component_match.a1.validate @@ -0,0 +1,103 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 17%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 34%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 51%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 60%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 74%] +........................................................................ [ 77%] +........................................................................ [ 80%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 94%] +........................................................................ [ 97%] +.................................................................. [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 225 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 224 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 317 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 100 0 100% +app/rag/retriever.py 234 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4221 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.md new file mode 100644 index 0000000..3191f1b --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.md @@ -0,0 +1,17 @@ +All green. Final report: + +**Implemented (task 02 — bounded name-hit bonus, D2/LOCKED A3)** +- `app/config.py`: `name_hit_bonus: float = 0.005` (`BOR_NAME_HIT_BONUS`) with validator (negative → `ValueError` naming the field; `0` = legal kill switch); `.env.example` entry next to `BOR_RECENCY_BOOST`. +- `app/rag/retriever.py`: new shared `_selection_order(chunks, bonus)` walk (first-seen-chunk rank semantics, best-cosine tracking, effective = best + bonus once per name-hit doc); `select_suggested`/`select_related`/`weak_hit_titles` gain `bonus: float | None = None` (settings default); dormant `select_documents*` untouched; module docstring updated. +- `scripts/eval_retrieval.py`: prints ` suggested: 1. src/path 2. …` per question (bonus included) + `name_hit_bonus=` in the header; smoke-tested live. +- Tests: config default/override/kill-switch/negative; retriever lift, non-lift past the gap, once-per-doc, golden pre-phase byte-identity (pinned list), exclusion under bonus, titles order, A8 inputs untouched (13 new tests). + +**Gates (all green)** +- `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` → 135 passed +- `uv run pytest --cov=app --cov-report=term-missing` → 2577 passed, TOTAL 99% (>90%) +- `uv run ruff check .` → clean; `uv run pyright` → 0 errors +- `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov` → 2 passed (neighbor suite unaffected) + +**Decisions**: re-sort is skipped entirely when `bonus==0` or no name-hit chunk exists — the LOCKED A3 byte-identical kill switch holds on any input; the D2 key `(−effective, −best_cosine, path, first_seen)` applies only when the bonus is active. No commits made (harness owns them). + +**Next pending task**: `03_suggested_folder_context.md` (D3 — suggested-folder lines in the HIGH prompt). diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.validate new file mode 100644 index 0000000..8cc9f8c --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__02_name_hit_bonus.a1.validate @@ -0,0 +1,104 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 34%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 51%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 99%] +....... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 225 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 231 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 317 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 100 0 100% +app/rag/retriever.py 242 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4236 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.md new file mode 100644 index 0000000..83e0614 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.md @@ -0,0 +1,19 @@ +All gates green. Final report: + +**Task 03 — suggested-folder context in the grounded prompt (phase 119, D3, LOCKED A4)** + +Implemented: +- `app/rag/agent.py`: new pure helper `suggested_folder_lines(db, suggested, max_lines=5, max_entries=8)` next to the `ls` helpers — one line per distinct parent folder (`//: …`, root `/: …`), `group_folder_listing` entry order (subfolders `name/ (N docs)`, singular `(1 doc)`, then files by relative filename), owning suggested doc excluded, ` +N more` with exact remainder +- `app/rag/prompts.py`: `build_high_prompt(..., folder_lines=())` — lines ride after `SUGGEST_INTRO`, before the first block, plain lines; empty ⇒ byte-identical phase-118 (pinned) +- `app/api/chat.py`: `plan_turn` gains `folder_lines` (HIGH branch only; LOW byte-identical); endpoint computes the same `select_suggested` walk + `suggested_folder_lines` before `plan_turn`; TurnPlan/durable record/log line untouched +- Tests: 8 helper tests (`test_agent.py`), 4 placement/byte-identity tests (`test_prompts.py`), 4 plan_turn/endpoint pass-through tests + `_FakeSession.execute` (`test_chat_gate.py`; same fake-session fix in `test_chat_cancel.py`, `test_embed_question_length.py`, `test_source_chip_quality.py`); re-cut the two full-HIGH-prompt pins in `tests/integration/test_kb_overview_api.py` to include the folder lines + +Verification: +- `uv run pytest tests/unit/test_agent.py tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v` → 223 passed +- `uv run pytest --cov=app --cov-report=term-missing` → 2543 passed, **99%** on `app/` +- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors +- Criterion check: 5 suggested docs spanning 3 folders ⇒ exactly 3 correctly-shaped lines (canonical README name lands in the prompt); `tests/e2e/test_summary_seed_context.py --no-cov` → 2 passed (mock unaffected) + +Notable decisions: subfolder entries are source-relative (`a/b/ (2 docs)`) to read exactly like the model's own `ls` output; file entries are basenames (the `//:` header supplies the folder). Empty-folder lines render header-only (`src/f/:`), the `ls` empty-level precedent. + +Next pending task: `.agents/phases/todo/119_name_signal_read_chips/04_cite_discipline_copy.md` diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.validate new file mode 100644 index 0000000..f5d35d4 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__03_suggested_folder_context.a1.validate @@ -0,0 +1,104 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 39%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 96%] +........................................................................ [ 99%] +....................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 226 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 231 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 347 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 102 0 100% +app/rag/retriever.py 242 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4269 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.md new file mode 100644 index 0000000..4c00196 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.md @@ -0,0 +1,14 @@ +Task 04 complete — all green. Report: + +**Implemented (phase-119 D4, LOCKED A5 — cite-discipline copy)** +- `app/rag/prompts.py`: `SUGGEST_INTRO` final sentence replaced verbatim with the LOCKED A5 discipline sentence (retires phase-118 "Cite the document(s) you used, by path."); constant docstring + `build_high_prompt` docstring updated (live confabulation closed, phase-118 sentence retired). +- `tests/unit/test_prompt_lock.py`: `SUGGEST_INTRO` anchors re-cut (sha256 `c6ffbb19…3302`, len 460; prefix anchor unchanged — only the final sentence moved); substring pin re-pointed + retirement `not in` assertion. `PERSONA`/`TOOLS_SECTION`/LOW anchors untouched, pass unchanged. +- `tests/unit/test_prompts.py`: new pins — HIGH prompt carries the sentence exactly once, inside `` after the start-here framing and before the first block (plain + folder-line builds; absent when no blocks); absent from all LOW builds; retired sentence gone. +- E2E audit: no e2e test referenced the old sentence (mock keys off markers, as expected). + +**Results** +- `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` → 56 passed +- `uv run pytest --cov=app --cov-report=term-missing` → 2545 passed, coverage 99% (gate >90%) +- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings + +**Deviations:** none. Next pending task: `.agents/phases/todo/119_name_signal_read_chips/05_chips_read_only.md` (D5 — `done.sources` = read docs only). diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.validate new file mode 100644 index 0000000..d1ae575 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__04_cite_discipline_copy.a1.validate @@ -0,0 +1,104 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 39%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 96%] +........................................................................ [ 99%] +......................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 226 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 231 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 347 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 102 0 100% +app/rag/retriever.py 242 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4269 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.md new file mode 100644 index 0000000..b0cdebc --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.md @@ -0,0 +1,10 @@ +All gates green. Final verification complete — the diff includes earlier phase-119 tasks' uncommitted work (harness makes the atomic commit at phase completion). + +**Report** + +- **Implemented (task 05, D5, LOCKED A1):** `app/api/chat.py` — done-frame citation walk now `holder.read_docs` only (deduped by `(source, path)`, read order); `plan.suggested_docs` removed from `cited_docs`; durable record (`query_log.sources` + per-turn log line = suggested + related + read) and `related` tier dedupe unchanged; deflected `[]` unchanged; module docstring, `TurnPlan.suggested_docs` comment, and done-section comments re-cut for A1 (phase-118 A4 union retired, zero-read chips nothing). +- **Pins re-targeted:** `test_chat_gate.py` (zero-read ⇒ `[]` + contrast vs A4 union; new multi-read read-order pin; new read-related-is-chip-not-nearby pin; `_CannedLLM` gained `read_paths`), `test_source_chip_quality.py` (2 A4-union pins), `test_chat_cancel.py` (completed-turn pin), integration: `test_chat_api.py` (10 spots), `test_auth_api.py` (2), `test_kb_overview_api.py` (`_cited_docs` → retrieval-reconstructed `_suggested_docs`). +- **Frontend:** audit only — `appendSources`/`addSources` already no-op on empty; no ≥1-chip assumptions; no change. +- **Results:** `uv run pytest tests/unit/test_chat_gate.py -v` → 58 passed; `uv run pytest --cov=app --cov-report=term-missing` → 2547 passed, app coverage 99% (>90%); `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings. +- **Notes:** E2E suites (mock LLM does read) still carry retired-A4 pins — re-targeting is task 06's explicit scope; left untouched. PLAN.md untouched per A6. +- **Next pending task:** `.agents/phases/todo/119_name_signal_read_chips/06_e2e_retarget_and_battery.md` diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.validate new file mode 100644 index 0000000..c6535d5 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__05_chips_read_only.a1.validate @@ -0,0 +1,104 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 39%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 81%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 96%] +........................................................................ [ 98%] +........................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 226 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 231 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 347 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 102 0 100% +app/rag/retriever.py 242 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4269 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.err b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.md b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.md new file mode 100644 index 0000000..95b16d3 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.md @@ -0,0 +1,9 @@ +**Task 06 complete — phase 119 finished (no pending tasks; 01–05 already in complete/, the pipeline gate moves 06).** + +1. Mock trigger: `repeat your folder map` + `` → answer ends with ` (folders: …)` echoing the folder lines verbatim (omitted when none); docstring section added; phrase unique repo-wide. +2. New suite `tests/e2e/test_name_signal_read_chips.py` (7-doc tracked fixture `tests/fixtures/namekb`, all docs name-hits so the +0.005 bonus is uniform; `forgejo-home` engineered LAST of top-5, probe-verified, pinned via the app's real `select_suggested`/`select_related` for all 4 questions): (a) tail echo proves `Source: namekb/forgejo/forgejo-home.md` last, (b) single-read → exactly one chip `zz-folder-map.md` + deduped related row, (c) plain question → `done.sources==[]` + zero chips, (d) folder line `namekb/forgejo/: forgejo-nginx.md` verbatim. **4 passed × 3 isolated runs** (determinism confirmed). +3. All retired A4 pins re-targeted (full audit of every `done["sources"]`/`source-chip` hit: read-set or zero, durable pins untouched); **24 retargeted suites green in isolation**; fixed one missed stored-sources pin (`test_sources_midstream_bug`) and hardened 3 settle waits against the chip-wait-removal race (`test_chat_rag` flake, `test_retrieval_quality`, `test_document_dates`). +4. Battery (replica restored via restore-test-db; fingerprint verified docs=1000, chunks=8866; alembic upgraded 0011→0020): **GATE PASS** — gitea README #4 (exact), forgejo 5/5 README #1 (exact), gateway README in top-5 #4 (overview #3, containment met), qwen3.8-27b quadlets #1/#2, Mongolia HIGH fts=5 (unchanged). +5. Real-model telemetry: **PASS** 25/25, 10/10 tool-turns, wall 141.8s; read turns fired round-1 `read` with zero drill-downs (D3 effect) — recorded in `TOOL_CALLING_TESTING.md` §11 (telemetry-only). +6. **Full output + verdict recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md`** — deviation: the task says paste into the task file, but the harness forbids editing task/phase files (harness wins on mechanics; intent preserved in the report). +7. Gates: `uv run pytest --cov=app` → **2547 passed, app 99%**; `uv run ruff check .` clean; `uv run pyright` 0/0/0. No git add/commit performed (pipeline gate commits per AGENTS.md rule 8). diff --git a/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.validate b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.validate new file mode 100644 index 0000000..c6535d5 --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/119_name_signal_read_chips__06_e2e_retarget_and_battery.a1.validate @@ -0,0 +1,104 @@ +........................................................................ [ 2%] +........................................................................ [ 5%] +........................................................................ [ 8%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 22%] +........................................................................ [ 25%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 39%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 81%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 96%] +........................................................................ [ 98%] +........................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 226 1 99% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 156 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 231 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 66 0 100% +app/models.py 128 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 347 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 206 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 219 3 99% +app/rag/llm.py 244 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 102 0 100% +app/rag/retriever.py 242 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 330 0 100% +-------------------------------------------------- +TOTAL 4269 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md b/.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md new file mode 100644 index 0000000..571eb9b --- /dev/null +++ b/.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md @@ -0,0 +1,130 @@ +# Task 06 record — battery run 2026-09-16 + real-model telemetry (phase 119, D6) + +**Deviation note:** the task says "paste the full output into this file" +(the task file), but the execution harness forbids editing task/phase +files. Per the harness conflict rule (harness wins on mechanics; the +intent is preserved), the full output + verdict are recorded HERE +(`.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md`) +and summarized in the final report. + +## Preconditions +- Replica restored via the `restore-test-db` skill (backup in + `/tmp/pre_restore_*.dump`): schema reset → single-transaction load of + `data/bor_live_kb_replica.dump.sql` (96 MB) → **fingerprint verified: + docs=1000, chunks=8866, + sources=deploy,homelab,homepage,ServMon,skills,vibe-bot, + alembic=0011** — exact 2026-09-05 snapshot match. +- Dump older than head (0011 vs 0020) → `uv run alembic upgrade head` + ran clean (0011 → 0020), per the skill's rule. +- Note (reported with the numbers, per the task): the replica predates + the phase-118 markdown-summary backfill (markdown `summary` NULL, no + markdown `is_summary` chunks) — the measured numbers below are the + same-state baseline the phase overview measured against. + +## Battery +`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` +(new tracked file: the three phase-119 name questions + the 2026-09-05 +incident regression + the deflection sanity). Full output: + +``` +eval: threshold=0.62 vector_candidates=100 lexical_candidates=30 rrf_k=60 recency_boost=0.0007 recency_half_life_days=365 name_hit_bonus=0.005 + +question: 'How do I deploy gitea?' + gate: best_cosine=0.8527 fts_hits=38 -> HIGH (answer) + 1. deploy/Deployments/reeseapps/gitea/app/playbook.yaml cosine=0.7640 fts=1 effective=0.03056 created=2026-09-16 (playbook) + 2. deploy/Deployments/reeseapps/gateway/README.md cosine=0.7737 fts=1 effective=0.02902 created=2026-09-16 (Proxy Server) + 3. homelab/active/container_gitea/gitea.md cosine=0.7610 fts=1 effective=0.02561 created=2026-09-16 (Gitea) + 4. homelab/active/kubernetes_gitea/gitea-values.yaml cosine=0.7232 fts=1 effective=0.02512 created=2026-09-16 (gitea-values) + 5. deploy/Deployments/reeseapps/gitea/app/gitea-postgres.env.j2 cosine=0.7158 fts=1 effective=0.02495 created=2026-09-16 (gitea-postgres.env) + suggested: 1. deploy/Deployments/reeseapps/gitea/app/playbook.yaml 2. deploy/Deployments/reeseapps/gitea/app/gitea-postgres.env.j2 3. deploy/Deployments/reeseapps/gateway/README.md 4. deploy/Deployments/reeseapps/gitea/README.md 5. deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2 + +question: 'How do I deploy forgejo?' + gate: best_cosine=0.7878 fts_hits=37 -> HIGH (answer) + 1. homelab/templates/container/foobar.md cosine=0.7878 fts=0 effective=0.01709 created=2026-09-16 (Podman foobar) + 2. deploy/Deployments/reeseapps/forgejo/README.md cosine=0.0000 fts=1 effective=0.01709 created=2026-09-16 (Reeseapps Forgejo) + 3. deploy/Deployments/reeseapps/README.md cosine=0.7857 fts=0 effective=0.01683 created=2026-09-16 (Deployments) + 4. deploy/Deployments/reeseapps/forgejo/app/forgejo-app.container cosine=0.0000 fts=1 effective=0.01683 created=2026-09-16 (forgejo-app) + 5. homelab/active/container_bifrost/bifrost.md cosine=0.7674 fts=0 effective=0.01657 created=2026-09-16 (Podman bifrost) + suggested: 1. deploy/Deployments/reeseapps/forgejo/README.md 2. deploy/Deployments/reeseapps/forgejo/app/forgejo-app.container 3. deploy/Deployments/reeseapps/forgejo/app/forgejo-app.env.j2 4. deploy/Deployments/reeseapps/forgejo/app/forgejo-postgres.container 5. deploy/Deployments/reeseapps/forgejo/app/forgejo-postgres.env.j2 + +question: 'How do I set up the gateway?' + gate: best_cosine=0.7815 fts_hits=37 -> HIGH (answer) + 1. homelab/active/device_shelly/shelly.md cosine=0.6940 fts=1 effective=0.02509 created=2026-09-16 (Shelly Devices) + 2. deploy/Deployments/reeseapps/forgejo/app/forgejo.network cosine=0.6726 fts=1 effective=0.02050 created=2026-09-16 (forgejo) + 3. homelab/active/device_framework_16/framework_16.md cosine=0.7815 fts=0 effective=0.01709 created=2026-09-16 (Framework Laptop 16) + 4. deploy/Deployments/delta/istio/gateway-metallb.yaml cosine=0.0000 fts=1 effective=0.01709 created=2026-09-16 (gateway-metallb) + 5. homelab/active/device_home_assistant/home_assistant.md cosine=0.7751 fts=0 effective=0.01683 created=2026-09-16 (Home Assistant) + suggested: 1. homelab/active/device_shelly/shelly.md 2. deploy/Deployments/delta/istio/gateway-metallb.yaml 3. deploy/Deployments/delta/istio/test-gateway.yaml 4. deploy/Deployments/reeseapps/gateway/README.md 5. deploy/Deployments/reeseapps/gateway/certbot/playbook.yaml + +question: 'Do you have notes on the qwen 3.8 27b model?' + gate: best_cosine=0.7484 fts_hits=31 -> HIGH (answer) + 1. deploy/Deployments/reeseapps/ai/deployments/epic/quadlets/qwen3.8-27b-epic-vulkan.container cosine=0.7401 fts=1 effective=0.03322 created=2026-09-16 (qwen3.8-27b-epic-vulkan) + 2. deploy/Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container cosine=0.7141 fts=1 effective=0.03058 created=2026-09-16 (qwen3.8-27b-juggernaut-vulkan) + 3. homelab/active/container_localai/localai.md cosine=0.6836 fts=1 effective=0.02570 created=2026-09-16 (Local AI with Anything LLM) + 4. homelab/active/software_ai_stack/ai_stack.md cosine=0.7027 fts=1 effective=0.02545 created=2026-09-16 (Self Hosted AI Stack) + 5. deploy/Deployments/reeseapps/ai/deployments/turbo/quadlets/qwen3.8-flash-next.container cosine=0.6674 fts=1 effective=0.02491 created=2026-09-16 (qwen3.8-flash-next) + suggested: 1. deploy/Deployments/reeseapps/ai/deployments/epic/quadlets/qwen3.8-27b-epic-vulkan.container 2. deploy/Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container 3. deploy/Deployments/reeseapps/ai/deployments/turbo/quadlets/qwen3.8-flash-next.container 4. homelab/active/container_localai/localai.md 5. homelab/active/software_ai_stack/ai_stack.md + +question: 'What is the capital of Mongolia?' + gate: best_cosine=0.6070 fts_hits=5 -> HIGH (answer) + 1. deploy/Deployments/reeseapps/README.md cosine=0.5589 fts=1 effective=0.02382 created=2026-09-16 (Deployments) + 2. homelab/active/container_nginx/nginx.md cosine=0.6070 fts=0 effective=0.01709 created=2026-09-16 (Ngnix) + 3. homelab/active/container_localai/localai.md cosine=0.0000 fts=1 effective=0.01709 created=2026-09-16 (Local AI with Anything LLM) + 4. homelab/active/os_arch/arch-workstation.md cosine=0.6057 fts=0 effective=0.01683 created=2026-09-16 (Workstation) + 5. homelab/active/software_k3s/k3s.md cosine=0.0000 fts=1 effective=0.01683 created=2026-09-16 (K3S) + suggested: 1. deploy/Deployments/reeseapps/README.md 2. homelab/active/container_nginx/nginx.md 3. homelab/active/container_localai/localai.md 4. homelab/active/os_arch/arch-workstation.md 5. homelab/active/software_k3s/k3s.md +``` + +**Verdict: GATE PASS.** +- gitea: canonical `deploy/Deployments/reeseapps/gitea/README.md` in + the suggested top-5 at **#4** — exact match with the overview's + post-phase numbers (playbook, gitea-postgres.env.j2, gateway README, + gitea README #4, gitea-web.env.j2). +- forgejo: **5/5 forgejo-tree, README #1** — exact match. +- gateway: canonical `deploy/Deployments/reeseapps/gateway/README.md` + in the suggested top-5 at **#4** (the overview measured #3 — same doc + family: shelly.md, the two istio gateway yamls, README, the certbot + playbook; the 0.005 bonus is an owner-tunable starting point, so the + battery records the realized margin — containment gate met). +- qwen 3.8 27b: the `qwen3.8-27b` quadlets stay in the suggested top-5 + (#1 epic, #2 juggernaut) — the digit-bearing prefix rule unchanged. +- Mongolia: verdict **HIGH, fts_hits=5** — unchanged from the pre-phase + replica run (the name-hit change moved nothing). + +## Real-model telemetry (NOT a gate — no tool-copy change) +`uv run python -m scripts.agent_realmodel_check --restore --mode +fixture`: **PASS** — 10/10 answered, caps=0, tool-turns=10/10, 25/25 +executed (100 %), contract 25/25 (100 %), wall 141.8 s (turbo). +Recorded in `TOOL_CALLING_TESTING.md` §11 (telemetry-only header; the +four conditions read under the phase-118 A7 semantics). Per-turn +reading: all four designed read turns (04/05/06/09) fired a single +round-1 `read` with no `ls` drill-downs (the D3 intended effect); +turn 08 flipped from a zero-call summary answer to a round-1 read; +listing turns drilled +4 calls total vs the phase-118 run; wall ++20.1 % (at the edge of the ~20 % band, just above the 97–135 s turbo +range — endpoint-load variance). + +## E2E +- New suite `tests/e2e/test_name_signal_read_chips.py` — 4 tests, + green in isolation, **run twice** (determinism: the mock's md5 + embeddings make the pinned tiers byte-stable): + `uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov` + → 4 passed (9.01 s), 4 passed (9.04 s), 4 passed (9.11 s, post-lint). +- Every retired-A4 pin re-targeted (audit rule applied to every + `done["sources"]` / `source-chip` hit); each retargeted suite green + in isolation (24 suites run individually, all passing; + `test_sources_midstream_bug::test_nav_switch_before_first_token_ + completes` had one missed stored-sources pin — fixed, re-run green). +- Two latent settle races surfaced by the chip-wait removal and fixed: + `test_chat_rag::test_chat_logs_query` (observed flake — the + query_log row is written before the done frame; the bubble-text wait + lands before it) and the weak `#send-btn` enabled waits in + `test_retrieval_quality` (the in-flight Stop control is enabled too) + → both now wait on the `#send-label` = "Send" recovery; + `test_document_dates` got the same hardening. + +## Gates +- `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, + app coverage 99 %** (>90 %). +- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 + errors, 0 warnings, 0 informations. diff --git a/.agents/screenshots/doc_modal_scrollbar_1280.png b/.agents/screenshots/doc_modal_scrollbar_1280.png index 1429699..fbb75f3 100644 Binary files a/.agents/screenshots/doc_modal_scrollbar_1280.png and b/.agents/screenshots/doc_modal_scrollbar_1280.png differ diff --git a/.env.example b/.env.example index ca0a2df..a01af94 100644 --- a/.env.example +++ b/.env.example @@ -61,6 +61,7 @@ BOR_HYBRID_LEXICAL_CANDIDATES=30 # FTS list width for the fusion BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant BOR_RECENCY_BOOST=0.0007 # max additive fused-score bonus a zero-age doc gets (phase 106); 0 = off (pre-phase ranking byte-identical) BOR_RECENCY_HALF_LIFE_DAYS=365 # recency decay timescale, days (phase 106); the boost decays by e^-1 (~0.37x) per timescale of age +BOR_NAME_HIT_BONUS=0.005 # selection-time bonus a name-hit doc's best fused score gets (phase 119); 0 = off (pre-phase order byte-identical) # --- Agent document tools (grounded turns may extend context: ls / read / grep) --- # BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools) diff --git a/TOOL_CALLING_TESTING.md b/TOOL_CALLING_TESTING.md index 6cc240c..bc47ed9 100644 --- a/TOOL_CALLING_TESTING.md +++ b/TOOL_CALLING_TESTING.md @@ -722,3 +722,84 @@ within the ~20 % band. The summary-seed behavior is proven against the real configured chat model — full text enters the context only through the capped `read` tool, and a summary-only answer is the intended fast path. + +## 11. Phase 119 — name-signal + read-chips TELEMETRY, 2026-09-16 +(Telemetry-only — NOT a re-triggered gate.) Phase 119 changed +retrieval (the component name-hit rule + the bounded name-hit bonus), +the grounded prompt (the suggested-folder context lines + the +cite-discipline sentence), and the citation surface (`done.sources` = +read docs only) — but the `AGENT_TOOLS` / `TOOLS_SECTION` tool copy is +BYTE-IDENTICAL (the phase-117 copy is untouched, an invariant of every +phase-119 task), so the tool-copy gate is NOT re-triggered. The +fixture battery was re-run against the real configured chat model as +TELEMETRY for the owner (task 06, D6); the four conditions are read +under the phase-118 A7 semantics (1/2/4 gated, 3 reported). + +**The verdict run.** `uv run python -m scripts.agent_realmodel_check +--restore --mode fixture` — the configured chat model (`turbo` per +`.env`), the fixture KB restored from the tracked dump, the real +grounded path, the full 10-question battery: + +``` +$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture +restore: ok in 0.04s (8 docs, 2 sources) +turn 01 | emitted=2 executed=2 cap=no defl=no | 17.10s | List the files in this directory. +turn 02 | emitted=4 executed=4 cap=no defl=no | 14.33s | List the documents you have in the … +turn 03 | emitted=9 executed=9 cap=no defl=no | 30.92s | List every document you have indexed. +turn 04 | emitted=1 executed=1 cap=no defl=no | 12.16s | Open the document … +turn 05 | emitted=1 executed=1 cap=no defl=no | 10.20s | Read … +turn 06 | emitted=1 executed=1 cap=no defl=no | 7.61s | Open the document … +turn 07 | emitted=2 executed=2 cap=no defl=no | 9.63s | Find the exact string "rbm-8842" in … +turn 08 | emitted=1 executed=1 cap=no defl=no | 15.02s | Which document has the title "Lab …" +turn 09 | emitted=1 executed=1 cap=no defl=no | 13.00s | What do you know about the qwen 3.8 … +turn 10 | emitted=3 executed=3 cap=no defl=no | 11.71s | List the files in the deployments … +gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=10 calls 25/25 executed (100%) contract 25/25 (100%) 2026-09-16 (wall 141.8s) +``` + +**The four conditions (read under the phase-118 A7 semantics) and +metrics.** + +| condition | gated under A7 | result | +|---|---|---| +| 1. all turns answer | yes | 10/10 answered — **GREEN** | +| 2. zero round-cap hits | yes | caps=0 — **GREEN** | +| 3. ≥6/10 turns emit ≥1 tool call | **no (reported)** | 10/10 tool-turns | +| 4. contract accuracy ≥ 0.90 | yes | 25/25 (100%) — **GREEN** | +| executed / emitted (reported) | no | 25/25 (100%) | + +Contract line: **contract 25/25 (100%)**. Wall time: **141.8 s**. +Model: **turbo** (the configured chat model). + +**Per-turn reading (the phase's intended latency effect, highlighted).** +All four designed read turns (04/05/06/09) each emitted exactly one +contract-correct `read` **in round 1** — no `ls` drill-downs at all +before the read: the suggested-folder context lines (phase 119, D3) +put the target files' names in the grounded prompt, which is exactly +the live-turn failure the phase fixes (the 2026-09-16 owner report: +the pre-phase gitea turn walked three `ls` levels — `deploy` → +`reeseapps` → `gitea` — before it could `read` the canonical README). +On this fixture battery the read targets were already seed-suggested, +so round-1 reads held from the phase-118 run — the drill-down +reduction shows instead on the real product-name questions (the +eval battery, recorded in the phase's task 06). Turn 08 (the title +lookup) flipped from the phase-118 run's zero-call summary answer to +a single round-1 `read` of `deployments/ansible/lab-inventory.md` — +a within-contract choice (the cite-discipline sentence, D4, licenses +citing a read suggested doc). The listing turns drilled slightly +deeper than the phase-118 run (01: 2 calls vs 1; 03: 9 vs 8; 07: +grep + a follow-up `read` = 2 vs 1) — +4 calls total, no caps. + +**Baseline comparison.** Against the phase-118 `turbo` run +(§10: 118.1 s wall, 21/21 calls, tool-turns 9/10): contract and +executed stay at 100 %, caps remain 0, tool-turns rise to 10/10, and +the wall time is 141.8 s (+20.1 %) — at the edge of the ~20 % band +the phase-94 gate used as its slowdown tripwire and just above the +97–135 s `turbo` range recorded in §3 (endpoint-load variance). As +telemetry this is within the normal band; no copy iteration was +triggered (the tool copy did not change). + +**Conclusion (telemetry).** All four conditions GREEN/reported-green +under the A7 semantics, contract 100 %, zero caps — the phase-119 +retrieval/prompt/citation changes did not degrade the real model's +tool-calling behavior, and the read turns confirm the D3 +intended effect: files named in the prompt are read in round 1. diff --git a/app/api/chat.py b/app/api/chat.py index 313a378..e7c31f3 100644 --- a/app/api/chat.py +++ b/app/api/chat.py @@ -55,12 +55,18 @@ point — and the LLM extends its context by ``read``-ing what it needs (the capped ``read`` tool is the ONLY full-text path). Both tiers are computed once per turn, for BOTH branches: ``suggested_docs`` (no floor, A3) and ``related_docs`` (rank 6+, the done frame's row). -``done.sources`` is the citation surface — suggested + agent-read, -deduped (A4: a grounded turn always shows chips); ``query_log.sources`` -and the per-turn log line keep recording the full retrieval -(suggested + related + read, LOCKED A3). The deflected branch's prompt -is byte-identical (weak-hit titles only — A8 untouched); its TurnPlan -still carries both tiers for the durable record. +Phase 119 (LOCKED A1, owner decision 2026-09-16 — the phase-118 A4 +union RETIRED): ``done.sources`` is the citation surface and carries +the AGENT-READ documents only (``holder.read_docs``, deduped by +``(source, path)``, read order) — a zero-read grounded turn (the +summary-only fast path) chips nothing, an accepted, owner-directed +consequence (the answer prose names the doc it used per the phase-119 +cite discipline, A5); ``query_log.sources`` and the per-turn log line +keep recording the full retrieval (suggested + related + read, 118-A3 +stands — the durable record records retrieval, not citations). The +deflected branch's prompt is byte-identical (weak-hit titles only — +A8 untouched); its TurnPlan still carries both tiers for the durable +record. KB overview (phase 31): the lite-generated outline of the knowledge base (single ``kb_overview`` row) is read per turn (one indexed PK @@ -198,6 +204,7 @@ from app.rag.agent import ( AgentHolder, MalformedReplyError, # phase 71: the recovery policy's terminal signal run_agent, + suggested_folder_lines, # phase 119 (D3): the suggested-folder context lines ) from app.rag.llm import ( EmbeddingError, @@ -282,8 +289,11 @@ class TurnPlan: deflected: bool system_prompt: str # The summary-seeded suggestion tier (phase 118, A6: top-N distinct - # documents, NO floor, A3 — the HIGH prompt carries their SUMMARIES; - # the done frame's citation surface + run_agent's seed_docs, A4). + # documents, NO floor, A3 — the HIGH prompt carries their SUMMARIES + # and they seed run_agent). Phase 119 (LOCKED A1): NOT the done + # frame's citation surface anymore — done.sources is the + # agent-read docs only (the phase-118 A4 union retired); this tier + # still rides the durable record (query_log + the log line). suggested_docs: list[Document] # Rank 6+ after the suggested set (phase 118: the next ranked docs # that are not already suggested, at most related_max_docs — the @@ -306,6 +316,7 @@ def plan_turn( settings: Settings, notes: Sequence[str] | None = None, kb_overview: str | None = None, + folder_lines: Sequence[str] = (), ) -> TurnPlan: """Apply the honesty gate (A8, revised) and assemble prompt + context. @@ -336,6 +347,17 @@ def plan_turn( prompts are byte-identical to the pre-phase text and ``kb_chars`` is 0. + *folder_lines* (phase 119, D3, LOCKED A4): the suggested-folder + context lines, computed by the chat endpoint (which holds the DB + session) via ``suggested_folder_lines`` over the SAME deterministic + suggested walk this function performs internally. The HIGH branch + passes them to ``build_high_prompt`` (they ride after + ``SUGGEST_INTRO``, before the first ```` block); the LOW + branch IGNORES them — the deflected prompt stays byte-identical. + Empty *folder_lines* ⇒ the HIGH prompt is byte-identical to the + phase-118 shape. The :class:`TurnPlan`, the durable record, and the + per-turn log line are untouched. + ``summary_hits`` (phase 30; phase 118 redefinition) counts the hit chunks with ``is_summary`` whose parent document is in the SUGGESTED set — both the HIGH and the LOW branch record it. @@ -380,7 +402,9 @@ def plan_turn( best_cosine, fts_hits, False, - build_high_prompt(suggested, notes=steering, kb_overview=kb_text), + build_high_prompt( + suggested, notes=steering, kb_overview=kb_text, folder_lines=folder_lines + ), suggested, related_docs, [], @@ -581,8 +605,26 @@ async def chat( kb_overview = load_kb_overview(step_db) with SessionLocal() as step_db: chunks = retrieve(step_db, request.message, question_vec) + # Phase 119 (D3, LOCKED A4): the suggested-folder + # context — the SAME deterministic suggested walk + # plan_turn performs internally (one extra cheap + # walk, no shared state); each suggested doc's + # parent-folder contents pre-seed the HIGH prompt + # so the model reads the right file in round 1 + # instead of walking ls drill-downs. The LOW branch + # ignores them (the deflected prompt stays + # byte-identical); empty ⇒ the phase-118 prompt is + # byte-identical too. + folder_lines = suggested_folder_lines( + step_db, + select_suggested(chunks, n=settings.suggested_docs), + ) plan = plan_turn( - chunks, settings, notes=steering_notes, kb_overview=kb_overview + chunks, + settings, + notes=steering_notes, + kb_overview=kb_overview, + folder_lines=folder_lines, ) except Exception: # noqa: BLE001 — DB failure mid-turn logger.exception( @@ -852,19 +894,25 @@ async def chat( # "start here" tier, no floor, A3) and the related # docs (``plan.related_docs``, rank 6+ after the # suggested set). The DURABLE record keeps the full - # retrieval (LOCKED A3: query_log records retrieval, + # retrieval (118-A3: query_log records retrieval, # not citations — even on deflected turns, where the - # weak hits are suggested). Phase 112/118: - # done.sources is the CITATION surface — it carries the - # suggested docs + the agent-read docs (deduped, - # LOCKED A4) on grounded turns and [] on deflected - # ones (a deflected answer cites nothing; the weak - # hits stay in the durable record). + # weak hits are suggested). Phase 112/119: + # done.sources is the CITATION surface — READ DOCS + # ONLY (phase 119, LOCKED A1, owner decision + # 2026-09-16; the phase-118 A4 suggested+read union + # is RETIRED): on grounded turns it carries + # ``holder.read_docs`` deduped by (source, path), + # read order — a zero-read grounded turn (the + # summary-only fast path) chips nothing, an accepted + # consequence (the answer prose names the doc per the + # phase-119 cite discipline, A5) — and [] on + # deflected ones (a deflected answer cites nothing; + # the weak hits stay in the durable record). # A cancelled turn (the generator closed by the # consumer) never reaches this step — no query_log row. cited_docs: list[Document] = [] cited_seen: set[tuple[str, str]] = set() - for doc in [*plan.suggested_docs, *holder.read_docs]: + for doc in holder.read_docs: key = (doc.source, doc.path) if key not in cited_seen: cited_seen.add(key) @@ -925,13 +973,15 @@ async def chat( # citations). The retrieval stays durably recorded above # (query_log.sources + the log line — observability # unchanged). - # Phase 118 (A3/A4): done.related carries the related - # tier — the ranked documents beyond the suggested set - # (rank 6+ after the contiguous top-N suggestion, capped - # by related_max_docs in the tiering), deduped against - # the cited list, the same (source, path) pattern as - # cited_docs: an agent-read related doc is a citation, - # never a "nearby doc". The UI renders it as the + # Phase 118 (A3) / phase 119 (A1): done.related carries + # the related tier — the ranked documents beyond the + # suggested set (rank 6+ after the contiguous top-N + # suggestion, capped by related_max_docs in the tiering), + # deduped against the cited list, the same (source, + # path) pattern as cited_docs — which is the READ DOCS + # only now: a read related doc is a chip, never a + # "nearby doc" (unchanged intent, keyed on read docs + # since phase 119). The UI renders the row as the # de-emphasized related-docs row, never a citation chip; # old clients ignore the field. cited_refs: list[SourceRef] = [] diff --git a/app/config.py b/app/config.py index 762eed7..23f56ac 100644 --- a/app/config.py +++ b/app/config.py @@ -292,6 +292,24 @@ class Settings(BaseSettings): #: of document age (full weight at age 0, ``weight/e`` at one #: half-life). ``<= 0`` fails startup loudly (same validator family). recency_half_life_days: int = 365 + #: Bounded name-hit bonus on the SELECTION-time document score + #: (phase 119, D2, LOCKED A3): the selection walks + #: (``select_suggested`` / ``select_related`` / ``weak_hit_titles`` + #: in ``app.rag.retriever``) add this to a document's best fused + #: chunk score when any of its chunks is a NAME HIT (the document + #: PATH matched a question name token under the D1 two-class rule) + #: — a product-name question ("How do I deploy gitea?") lifts the + #: product's own documents into the seeded suggestion tier. The + #: phase-106 recency-boost precedent: additive, bounded, single + #: apply site (the selection layer ONLY — chunk scores, ``fuse()``, + #: ``retrieve()``, the A8 honesty gate, and ``query_log.top_score`` + #: are untouched), ``0`` = off (the pre-phase walk returns + #: byte-identical — the kill switch); a negative value fails startup + #: loudly. 0.005 ≈ a 2-4 rank head start on the k=60 RRF scale + #: (rank 1 vs 5 in one list differs by ~0.0010) — an owner-tunable + #: starting point, not a calibrated constant (the phase-119 battery + #: records the realized margins). + name_hit_bonus: float = 0.005 # --- Admin & sign-in (phase 16; A10 revised 2026-08-22) --- # Single-admin auth via a signed session cookie (Starlette @@ -546,6 +564,17 @@ class Settings(BaseSettings): raise ValueError("recency_half_life_days must be > 0 (days)") return v + @field_validator("name_hit_bonus") + @classmethod + def _name_hit_bonus_non_negative(cls, v: float) -> float: + """``0`` is the byte-identical kill switch (the pre-phase + selection order) — a NEGATIVE bonus would demote name-hit + documents, the exact opposite of D2 (the ``agent_max_rounds`` + pattern, phase 119).""" + if v < 0: + raise ValueError("name_hit_bonus must be >= 0 (0 = off)") + return v + @field_validator("db_pool_size") @classmethod def _db_pool_size_positive(cls, v: int) -> int: diff --git a/app/rag/agent.py b/app/rag/agent.py index 074f4bf..12a1162 100644 --- a/app/rag/agent.py +++ b/app/rag/agent.py @@ -920,6 +920,98 @@ def render_folder_listing( return "\n".join([header, *body]) +def suggested_folder_lines( + db: Session, + suggested: Sequence[Document], + max_lines: int = 5, + max_entries: int = 8, +) -> list[str]: + """The HIGH prompt's suggested-folder context lines (phase 119, D3, + LOCKED A4) — pure composition over the existing ``ls`` machinery. + + One line per DISTINCT parent folder of the *suggested* documents — + in suggested-doc order, deduped by ``(source, parent prefix)`` (the + first suggested doc wins the slot), at most *max_lines* lines: + + * the parent prefix is the path up to (excluding) the last ``/`` + (``""`` = the source root); + * the line is ``//: e1, e2, …`` (the source root + renders as ``/: …`` — the filesystem-style folder path, + trailing slash included, + the colon) with the folder's direct + children in the + EXISTING ``ls`` folder-level rendering order — the direct + subfolders first (``name/ (N docs)``, the recursive doc count, + singular ``(1 doc)``), then the files by relative filename — so + the line reads the same as the model's own ``ls`` output of that + folder (the :func:`group_folder_listing` grouping, over + :func:`_source_document_rows` + :func:`_source_folder_summaries`); + * the suggested document that OWNS the line is excluded from the + entries (its identity is already in its ```` block — the + line is the folder's OTHER contents, the pre-seed that makes the + model ``read`` the right file in round 1 instead of walking the + ``ls`` drill-downs); + * at most *max_entries* entries, then `` +N more`` (N = the + remaining count, the true pre-cap folder total — the suggested + doc leaves the total even when its row sat past + :data:`LS_MAX_FILE_LINES`); a folder whose only entry was the + suggested doc renders its header alone (the ``… — 0 documents, 0 + folders:`` empty-level precedent). + + Empty *suggested* → ``[]`` (the caller then builds the + byte-identical phase-118 prompt). Module-level so unit tests can + monkeypatch the fetchers without a database (the house style: + :func:`ls_top` / :func:`ls_folder` compose the same fetchers). + """ + lines: list[str] = [] + seen: set[tuple[str, str]] = set() + rows_cache: dict[str, tuple[list[tuple[str, str, str]], dict[str, str]]] = {} + for doc in suggested: + if len(lines) >= max_lines: + break + prefix = folder_of(doc.path) + key = (doc.source, prefix) + if key in seen: + continue + seen.add(key) + if doc.source not in rows_cache: + rows_cache[doc.source] = ( + _source_document_rows(db, doc.source), + _source_folder_summaries(db, doc.source), + ) + rows, summaries = rows_cache[doc.source] + subfolders, files, total_files = group_folder_listing( + doc.source, prefix, rows, summaries + ) + entries = [ + f"{sub}/ ({count} {'doc' if count == 1 else 'docs'})" + for sub, count, _summary in subfolders + ] + # The owning suggested doc is a direct file of this folder — + # drop it from the entries (its identity is already in its + # block); its count leaves the total either way + # (even when its row sat past the LS_MAX_FILE_LINES file cap). + files = [entry for entry in files if entry[1] != doc.path] + # The file entries come AFTER the subfolders (the ls folder-level + # order) and ride by RELATIVE filename — the basename within the + # folder (the line's ``//:`` header supplies the + # folder; combined ``source/prefix/name`` is the read identity). + entries.extend(path.rsplit("/", 1)[-1] for _src, path, _title, _date in files) + total = len(subfolders) + total_files - 1 + shown = entries[:max_entries] + # The pinned identity shape: ``//:`` (the source + # root: ``/:``) — the filesystem-style folder path + # (trailing slash included) + the colon. + identity = f"{doc.source}/{prefix}/" if prefix else f"{doc.source}/" + line = f"{identity}:" + if shown: + suffix = ", ".join(shown) + if total > len(shown): + suffix += f" +{total - len(shown)} more" + line += f" {suffix}" + lines.append(line) + return lines + + def find_document(db: Session, source: str, path: str) -> Document | None: """The indexed document at ``(source, path)``, or ``None``. diff --git a/app/rag/prompts.py b/app/rag/prompts.py index 54c3355..23e1bda 100644 --- a/app/rag/prompts.py +++ b/app/rag/prompts.py @@ -132,21 +132,30 @@ _KB_INTRO = ( ) #: The ```` section's start-here intro (phase 118, task 03 — -#: the owner directive, TODO L3): the seeded blocks are the SUMMARIES of -#: the top-ranked documents for the question, opt-in starting points — -#: start here if one seems right; ``read`` the document's combined -#: ``source/path`` to add its FULL content before answering specifics -#: (the full text is NOT in the prompt until you read it); cite the -#: document(s) you used, by path. Rendered BEFORE the first block, only -#: when at least one block is present. The ```` tag and the -#: per-block ```` markup stay byte-stable around it (the E2E -#: mock's block parser and the ``read``-tool copy key off both). +#: the owner directive, TODO L3; phase 119, task 04, LOCKED A5 — the +#: final sentence re-cut to the cite discipline): the seeded blocks are +#: the SUMMARIES of the top-ranked documents for the question, opt-in +#: starting points — start here if one seems right; ``read`` the +#: document's combined ``source/path`` to add its FULL content before +#: answering specifics (the full text is NOT in the prompt until you +#: read it); cite ONLY the documents you read (or the suggested document +#: you answered from without reading it) — never one you neither read +#: nor used. The discipline sentence closes the phase-119 live +#: confabulation — the model's "Docs used:" line cited a file it never +#: read, confabulated from the suggestion blocks sitting in context — +#: and retires the phase-118 "cite the document(s) you used, by path" +#: sentence. Rendered BEFORE the first block, only when at least one +#: block is present. The ```` tag and the per-block +#: ```` markup stay byte-stable around it (the E2E mock's +#: block parser and the ``read``-tool copy key off both). SUGGEST_INTRO = ( "The blocks below are the summaries of the top-ranked documents for " "your question — start here if one seems right to you: call `read` " "with that document's combined `source/path` to add its full content " "before answering specifics (its full text is not in the prompt until " - "you read it). Cite the document(s) you used, by path." + "you read it). Cite only the document(s) you read — or, if you " + "answered from a suggested summary without reading it, cite that " + "suggested document — never a document you neither read nor used." ) #: The ```` instructions section — **HIGH prompt only** (phase 37, @@ -416,6 +425,7 @@ def build_high_prompt( documents: Sequence[Document], notes: Sequence[str] | None = None, kb_overview: str | None = None, + folder_lines: Sequence[str] = (), ) -> str: """Grounded turn: locked persona (+ steering, + KB overview) + the top-ranked documents' SUMMARY blocks + the ```` instructions @@ -423,7 +433,8 @@ def build_high_prompt( ``grep`` shapes; phase 72: the document-identity contract up front; phase 118, LOCKED A6 re-revising A7: the ```` section seeds SUMMARIES, never full texts — full text enters the context - only through the capped ``read`` tool). + only through the capped ``read`` tool; phase 119, LOCKED A4: the + suggested-folder context lines, below). Section order: ```` → ```` → ```` → ```` → ````; empty steering/overview omit their @@ -435,11 +446,28 @@ def build_high_prompt( start-here :data:`SUGGEST_INTRO` line (before the first block — the phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the blocks are the summaries of the top-ranked documents, ``read`` adds - the full text, and the answer cites the document(s) used by path. - This is NOT the reverted phase-72 in-context reminder (that copy + the full text, and the answer cites only the documents it read (or + the suggested document it answered from without reading — the + phase-119 cite discipline, LOCKED A5; the phase-118 "cite what you + used, by path" sentence is retired). This is NOT the reverted + phase-72 in-context reminder (that copy taught the seed documents as already-read context; this one teaches the summary-as-starting-point contract the A6 re-revision requires). + *folder_lines* (phase 119, D3, LOCKED A4): the suggested-folder + context lines (``app.rag.agent.suggested_folder_lines``) — the + direct children of each suggested document's parent folder, so the + model ``read``s the right file in round 1 instead of walking the + ``ls`` drill-downs. When blocks are present AND the lines are + non-empty, the ```` body is ``SUGGEST_INTRO``, then the + folder lines (each on its own line, immediately after the intro + line), a blank line, then the first ```` block. They are + PLAIN lines — no new markup/tag (the E2E mock keys off the + ```` marker and the LAST block's tail, so the lines must + never land after a summary). Empty *folder_lines* ⇒ the phase-118 + build is byte-identical (pinned in + ``tests/unit/test_prompts.py``). + Each ```` block carries the identity attributes ``source`` / ``path`` / ``title`` — and, since phase 106 (D5), ``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``, @@ -466,7 +494,14 @@ def build_high_prompt( "(no documents matched — do not invent specifics)" ) if blocks: - body = SUGGEST_INTRO + "\n\n" + body + if folder_lines: + # Phase 119 (D3, LOCKED A4): the folder lines ride immediately + # after the intro line, one per line, a blank line, then the + # first block — plain lines, no new markup (the + # block markup AND body stay byte-identical around them). + body = SUGGEST_INTRO + "\n" + "\n".join(folder_lines) + "\n\n" + body + else: + body = SUGGEST_INTRO + "\n\n" + body prompt = _base("HIGH") for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])): if part: diff --git a/app/rag/retriever.py b/app/rag/retriever.py index b93c8c4..5bb5e92 100644 --- a/app/rag/retriever.py +++ b/app/rag/retriever.py @@ -4,16 +4,30 @@ each carrying its cosine ``1 − distance`` (the honesty-gate input). * **Lexical list** — top-N chunks matching an OR-``tsquery`` over the question's tokens, ordered by ``ts_rank``, UNION the name-hit list: - documents whose normalized name (title + path stem, alnum-only, - lowercased) contains a DIGIT-BEARING normalized question token of - length >= 4 — bare tokens (``1panel``) and the join of adjacent - question tokens starting with a letter ("Qwen 3.8" → ``qwen38``). - The digit requirement is the precision guard: plain prose words - ("server", "arguments" — 4+ chars but no digit) never name-match, - while versioned product names (the incident's whole point) always - carry one. The name-hit documents LEAD the lexical list (ranked by - match count, then total matched length, then catalog order; capped - at :data:`NAME_HIT_LIMIT`), + documents whose PATH components match a question name token under + the two-class rule (the 2026-09-05 incident fix, extended + 2026-09-16 for product names WITHOUT digits — phase 119, LOCKED + A2). The token set is class-agnostic — every normalized whitespace + token of length >= 4 (dotted kept whole: ``llama.cpp`` → + ``llamacpp``) plus the versioned join ("Qwen 3.8" → ``qwen38``) — + the digit distinction lives on the MATCH side: + + * a DIGIT-BEARING token prefix-matches a normalized path part or + file stem (``qwen38`` → + ``qwen3.8-27b-juggernaut-vulkan.container`` — the incident's + original precision guard); + * a DIGITLESS token exact-matches a normalized path part, file + stem, or stem sub-component (``gitea`` → the ``gitea/`` folder, + ``gitea.md``, ``kubernetes_gitea``, ``gitea-values``). + + TITLES ARE NEVER MATCHED — titles are prose: ``deploy/Deployments/ + reeseapps/README.md`` is titled "Deployments" and must not + name-match the common token "deploy" (the owner-verified failure + mode of the naive relaxation). The name-hit documents LEAD the + lexical list (ranked by distinct matched-token count, then catalog + order ``(source, path)`` — the old total-matched-length tie-break + is retired, it outranked 5-char product names by 6-char common + tokens; capped at :data:`NAME_HIT_LIMIT`), the FTS rows follow. This is what finds name-your-tool questions ("gitlab") that vector similarity buries — and, the 2026-09-05 incident, the versioned-name case the default parser lexes @@ -23,7 +37,8 @@ ``llama``/``cpp`` tokens). The name-hit rows carry ``fts_hit=True`` (they ARE the lexical signal — the A8 honesty gate answers on them only when the best cosine clears ``BOR_LEXICAL_SUPPORT_FLOOR``, - A8 revised 2026-09-14) and ``cosine=0.0``; the RRF fusion is + A8 revised 2026-09-14), ``cosine=0.0``, and ``name_hit=True`` (the + selection-tier bonus input, phase 119 D2); the RRF fusion is unchanged (same lists, same ``1/(k+rank)`` terms). * **Fusion** — Reciprocal Rank Fusion (``score = Σ 1/(k + rank)`` over the lists a chunk appears in; chunks hit by both lists get both terms). The @@ -35,6 +50,19 @@ ranked documents (up to ``BOR_RELATED_MAX_DOCS``) become the related tier. +Phase 119, D2 (LOCKED A3): the SELECTION walks (``select_suggested``, +``select_related``, ``weak_hit_titles``) re-rank on an EFFECTIVE score +— the document's best fused chunk score plus ``BOR_NAME_HIT_BONUS`` +when any of its chunks is a name hit (the D1 path match). A +product-name question ("How do I deploy gitea?") thus lifts the +product's own documents into the seeded suggestion tier even when the +name hit only LEADS the lexical list. The bonus lives in the selection +layer only (the phase-106 recency-boost pattern: additive, bounded, +single apply site): chunk scores, the fusion, ``fuse()``, +:func:`retrieve()`, the A8 honesty gate, and ``query_log.top_score`` +are untouched; ``0`` reproduces the pre-phase walk byte-identically +(the kill switch) and a negative value fails startup loudly. + The product requirement (A7, re-revised by the phase-118 owner directive, LOCKED A6, 2026-09-15): the retrieval path seeds **summaries** — the suggestion tier (:func:`select_suggested`, top-N distinct documents, no @@ -96,6 +124,10 @@ NAME_HIT_LIMIT = 10 #: Alphanumeric runs of a lowercased string (name normalization). _ALNUM_RE = re.compile(r"[a-z0-9]+") +#: Non-alphanumeric runs of a lowercased string (the stem sub-component +#: split — ``kubernetes_gitea`` → ``kubernetes`` / ``gitea``). +_STEM_SPLIT_RE = re.compile(r"[^a-z0-9]+") + def _normalize_name(s: str) -> str: """Lowercased, alnum-only form of *s* (``Qwen 3.8`` → ``qwen38``).""" @@ -103,14 +135,23 @@ def _normalize_name(s: str) -> str: def name_hit_tokens(question: str) -> list[str]: - """The normalized name-match candidates of one question. + """The name-match candidates of one question (class-agnostic, + LOCKED A2). - Only DIGIT-BEARING candidates count (the precision guard — a plain - prose word like "server" or "arguments" must never name-match a - document; a versioned product name always carries a digit): + Every qualifying whitespace token is a candidate — the digit + distinction (which CLASS of match a token gets) lives on the + matching side (:func:`_name_hit_chunks`), because product names + WITHOUT digits ("gitea", "forgejo", "gateway") were the + 2026-09-16 live finding: the old digit-only list gave them no + name signal at all, so an OR-tsquery dominated by a common token + ("deploy") buried the product's own documents. Prose precision is + now carried by the match class itself (a digitless token must + EQUAL a whole path component — "server"/"arguments" rarely do), + not by filtering the candidate list. * the :func:`_normalize_name` form of every whitespace token, kept - when at least :data:`NAME_TOKEN_MIN_LEN` chars (``1panel``, + when at least :data:`NAME_TOKEN_MIN_LEN` chars — dotted tokens + kept whole (``llama.cpp`` → ``llamacpp``, ``1panel``, ``qwen38`` from a single written token); * the versioned-name JOIN — the normalized concatenation of every ADJACENT token pair whose SECOND token is purely numeric (a @@ -121,18 +162,17 @@ def name_hit_tokens(question: str) -> list[str]: starts with the first token's text (a letter in practice), so no digit-leading artifact (``38show``) can survive. - Order of first appearance, de-duplicated. + Order of first appearance, de-duplicated. Matching applies the + A2 rule: digit-bearing candidates prefix a normalized path part + or file stem; digitless candidates equal a part, stem, or stem + sub-component — titles are never matched. """ tokens = question.split() out: list[str] = [] seen: set[str] = set() def add(tok: str) -> None: - if ( - len(tok) >= NAME_TOKEN_MIN_LEN - and any(ch.isdigit() for ch in tok) - and tok not in seen - ): + if len(tok) >= NAME_TOKEN_MIN_LEN and tok not in seen: seen.add(tok) out.append(tok) @@ -187,6 +227,13 @@ class RetrievedChunk: resolves to the full source document through the unchanged chunk→document mapping (A7 revised). Default ``False`` keeps every ordinary content chunk valid. + * ``name_hit`` — True for the name-hit representative row (phase 119, + D1): the chunk came from the document PATH match, not the + OR-tsquery — the selection tier's bonus input (task 02, D2). + Default ``False`` keeps every ordinary construction valid; + :func:`fuse`'s ``replace()`` copies it (the double-hit merge ORs it + in — a vector row that is also the name hit's representative chunk + keeps the flag). """ chunk_id: uuid.UUID @@ -197,6 +244,7 @@ class RetrievedChunk: cosine: float = 0.0 fts_hit: bool = False is_summary: bool = False + name_hit: bool = False def lexical_tsquery(question: str) -> str | None: @@ -234,7 +282,9 @@ def fuse( Lexical-only hits (no vector rank) enter with ``cosine=0.0`` and ``fts_hit=True``; vector chunks matched by the lexical list get ``fts_hit=True`` in place (the input objects are mutated — callers - should not reuse them afterwards). + should not reuse them afterwards), and ``name_hit=True`` is ORed in + (a vector row that is also the name hit's representative chunk is a + name-hit row — the phase-119 selection tier must see it). """ if k <= 0: raise ValueError("rrf k must be > 0") @@ -247,7 +297,9 @@ def fuse( term = 1.0 / (k + rank) if rc.chunk_id in by_id: existing = by_id[rc.chunk_id] - by_id[rc.chunk_id] = replace(existing, fts_hit=True) + by_id[rc.chunk_id] = replace( + existing, fts_hit=True, name_hit=existing.name_hit or rc.name_hit + ) fused[rc.chunk_id] += term else: rc = replace(rc, fts_hit=True) @@ -390,23 +442,76 @@ _NAME_HIT_SQL = text( ) -def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]: - """The documents whose NAME matches the question (the 2026-09-05 - incident, the versioned-name signal). +def _name_parts(path: str) -> tuple[set[str], set[str]]: + """The normalized name components of a document path (LOCKED A2). - A document is a name hit when its normalized name — - :func:`_normalize_name` of its ``title`` followed by the normalized - path stem (``qwen3.8-27b-juggernaut-vulkan.container`` → - ``qwen3827bjuggernautvulkan``) — contains at least one - :func:`name_hit_tokens` candidate (``qwen38``). Ranked by - (distinct matched tokens, total matched length, source, path) — - catalog order is the final deterministic tie-break — capped at - :data:`NAME_HIT_LIMIT`. Each hit becomes one lexical - :class:`RetrievedChunk` (its representative chunk, ``fts_hit=True``, - ``cosine=0.0``). Two lightweight queries: one projection over - (id, source, path, title) in catalog order, one LATERAL chunk - fetch for the ≤ :data:`NAME_HIT_LIMIT` winners (no full-content - load; the content joins in via the row fetch below). + Returns ``(prefix_set, equal_set)``: + + * ``prefix_set`` — the normalized form of every path part plus the + normalized file stem (``qwen3.8-27b-juggernaut-vulkan.container`` + → ``{…, qwen3827bjuggernautvulkancontainer, qwen3827bjuggernaut…}``): + DIGIT-BEARING tokens prefix-match these; + * ``equal_set`` — ``prefix_set`` plus the stem's sub-components + (the stem lowercased, split on non-alphanumeric runs, each piece + normalized, empties dropped: ``kubernetes_gitea`` → + ``kubernetes`` / ``gitea``): DIGITLESS tokens exact-match these. + """ + p = Path(path) + prefix = {_normalize_name(part) for part in p.parts} + prefix.discard("") + prefix.add(_normalize_name(p.stem)) + equal = set(prefix) + for piece in _STEM_SPLIT_RE.split(p.stem.lower()): + norm = _normalize_name(piece) + if norm: + equal.add(norm) + return prefix, equal + + +def _name_token_matches(token: str, prefix: set[str], equal: set[str]) -> bool: + """The two-class match of one candidate token against one path + (LOCKED A2, phase 119): + + * the token CONTAINS A DIGIT → it is a PREFIX of a normalized path + part or file stem (``qwen38`` → + ``qwen3.8-27b-juggernaut-vulkan.container``); + * the token HAS NO DIGIT → it EQUALS a normalized path part, file + stem, or stem sub-component (``gitea`` → the ``gitea/`` folder, + ``gitea.md``, ``kubernetes_gitea``, ``gitea-values``). + """ + if any(ch.isdigit() for ch in token): + return any(part.startswith(token) for part in prefix) + return token in equal + + +def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]: + """The documents whose PATH matches the question (the 2026-09-05 + incident's versioned-name signal, extended 2026-09-16 for product + names without digits — phase 119, D1, LOCKED A2). + + A document is a name hit when at least one :func:`name_hit_tokens` + candidate matches its path components under the two-class rule + (:func:`_name_token_matches`): DIGIT-BEARING tokens prefix-match a + normalized path part or file stem (``qwen38`` → + ``qwen3.8-27b-juggernaut-vulkan.container``); DIGITLESS tokens + exact-match a normalized path part, file stem, or stem + sub-component (``gitea`` → the ``gitea/`` folder, ``gitea.md``, + ``kubernetes_gitea``, ``gitea-values``). **Titles are never + matched** — titles are prose: ``deploy/Deployments/reeseapps/ + README.md`` is titled "Deployments" and must NOT name-match the + common token ``deploy`` or the 9 other deployment-titled docs + (the owner-verified failure mode of the naive title relaxation). + + Ranked by (distinct matched-token count DESC, then ``(source, + path)`` catalog order) — the old total-matched-length tie-break is + RETIRED (it systematically outranked 5-char product names by + 6-char common tokens) — capped at :data:`NAME_HIT_LIMIT`. Each hit + becomes one lexical :class:`RetrievedChunk` (its representative + chunk, ``fts_hit=True``, ``cosine=0.0``, ``name_hit=True``). Two + lightweight queries: one projection over (id, source, path, title) + in catalog order (the title is selected but never matched), one + LATERAL chunk fetch for the ≤ :data:`NAME_HIT_LIMIT` winners (no + full-content load; the content joins in via the row fetch below). """ tokens = name_hit_tokens(question) if not tokens: @@ -416,22 +521,22 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]: Document.source, Document.path ) ).all() - scored: list[tuple[int, int, uuid.UUID]] = [] + scored: list[tuple[int, uuid.UUID]] = [] by_id: dict[uuid.UUID, tuple[str, str]] = {} # id -> (source, path) - for doc_id, source, path, title in rows: + for doc_id, source, path, _title in rows: by_id[doc_id] = (source, path) - name = _normalize_name(title) + _normalize_name(Path(path).stem) - matched = [t for t in tokens if t in name] + prefix, equal = _name_parts(path) + matched = sum(1 for t in tokens if _name_token_matches(t, prefix, equal)) if matched: - scored.append((len(matched), sum(len(t) for t in matched), doc_id)) + scored.append((matched, doc_id)) if not scored: return [] - # Ranked by (distinct matched tokens, total matched length), the - # deterministic catalog tie-break (source, path) last. - scored.sort( - key=lambda s: (-s[0], -s[1], by_id[s[2]][0], by_id[s[2]][1]) - ) - ids = [s[2] for s in scored[:NAME_HIT_LIMIT]] + # Ranked by (distinct matched-token count DESC), the deterministic + # catalog tie-break (source, path) — the old total-matched-length + # tie-break is retired (it outranked 5-char product names by + # 6-char common tokens). + scored.sort(key=lambda s: (-s[0], by_id[s[1]][0], by_id[s[1]][1])) + ids = [s[1] for s in scored[:NAME_HIT_LIMIT]] hit_rows = list(db.execute(_NAME_HIT_SQL, {"ids": ids}).all()) # The LATERAL query returns winners in id order; re-order by the # ranked order computed above so the lexical list is deterministic. @@ -460,6 +565,7 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]: cosine=0.0, # no vector rank — name-only hit fts_hit=True, # lexical signal — A8 answers if cosine corroborates is_summary=bool(row.is_summary), + name_hit=True, # phase 119 — the selection tier's bonus input ) ) return out @@ -556,20 +662,92 @@ def retrieve( return fused -def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]: - """Distinct parent-document titles of *chunks*, best fused score first. +def _selection_order( + chunks: Sequence[RetrievedChunk], + bonus: float, +) -> list[tuple[Document, float, float, int]]: + """The shared selection walk of the phase-119 name-hit bonus (D2, + LOCKED A3) — one walk for ``select_suggested``, ``select_related``, + and ``weak_hit_titles``. + + Returns ``(document, effective_score, best_cosine, first_seen_index)`` + for each distinct document, where *effective_score* is the + document's best fused chunk score plus *bonus* when ANY of its + chunks carries ``name_hit`` (the D1 path match — the bonus is per + DOCUMENT, applied ONCE no matter how many of the document's chunks + are name hits). + + The walk keeps the EXISTING selection semantics (not just the loop): + the same stable score-descending order as + :func:`select_documents_tiered` / :func:`select_suggested` — a + document's rank position is fixed by its FIRST seen chunk — with its + best cosine tracked across ALL of its chunks (the tiered walk's + tracking). + + * *bonus* ``== 0`` (the kill switch) or no name-hit chunk present: + the document order is IDENTICAL to the pre-phase walk — no re-sort + happens at all (byte-identical, LOCKED A3); + * otherwise the documents are ordered by + ``(−effective_score, −best_cosine, document.path, + first_seen_index)`` — the bounded re-rank: a name-hit document + gets a head start on the fused scale, and an effective-score tie + resolves by cosine, then path, then the pre-bonus rank. + + The bonus lives in the SELECTION layer only: the chunk objects are + never modified — their ``score``/``cosine``/``fts_hit`` (the A8 + gate's inputs) and ``query_log.top_score`` are untouched. + """ + order: list[Document] = [] + best_score: dict[uuid.UUID, float] = {} + best_cosine: dict[uuid.UUID, float] = {} + any_name_hit: dict[uuid.UUID, bool] = {} + first_seen: dict[uuid.UUID, int] = {} + for idx, rc in enumerate(sorted(chunks, key=lambda c: c.score, reverse=True)): + doc = rc.document + if doc.id in best_cosine: + if rc.cosine > best_cosine[doc.id]: + best_cosine[doc.id] = rc.cosine + if rc.name_hit: + any_name_hit[doc.id] = True + else: + order.append(doc) + best_score[doc.id] = rc.score + best_cosine[doc.id] = rc.cosine + any_name_hit[doc.id] = rc.name_hit + first_seen[doc.id] = idx + effective = { + doc.id: best_score[doc.id] + (bonus if any_name_hit[doc.id] else 0.0) + for doc in order + } + if bonus > 0.0 and any(any_name_hit[doc.id] for doc in order): + order.sort( + key=lambda doc: ( + -effective[doc.id], + -best_cosine[doc.id], + doc.path, + first_seen[doc.id], + ) + ) + return [ + (doc, effective[doc.id], best_cosine[doc.id], first_seen[doc.id]) + for doc in order + ] + + +def weak_hit_titles( + chunks: Sequence[RetrievedChunk], + bonus: float | None = None, +) -> list[str]: + """Distinct parent-document titles of *chunks*, best SELECTION score + first (the phase-119 name-hit bonus applied — see + :func:`_selection_order`, D2, LOCKED A3). Deflection mode (PLAN §6, A8) is built from these *titles only* — the LOW prompt and the "Maybe try" chips never see document content. """ - titles: list[str] = [] - seen: set[uuid.UUID] = set() - for rc in sorted(chunks, key=lambda c: c.score, reverse=True): - if rc.document.id in seen: - continue - seen.add(rc.document.id) - titles.append(rc.document.title) - return titles + if bonus is None: + bonus = get_settings().name_hit_bonus + return [doc.title for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)] def select_documents_tiered( @@ -685,54 +863,56 @@ def select_documents( def select_suggested( chunks: Sequence[RetrievedChunk], n: int | None = None, + bonus: float | None = None, ) -> list[Document]: - """Top-N distinct parent documents in fused rank order — the phase-118 - "start here" suggestion tier (LOCKED A3), with NO cosine floor. + """Top-N distinct parent documents in SELECTION rank order — the + phase-118 "start here" suggestion tier (LOCKED A3, re-revised by + phase 119: the walk now carries the bounded name-hit bonus, D2), + with NO cosine floor. Distinct parent documents are walked in the SAME stable score- descending order as :func:`select_documents_tiered` (a document's rank - position is fixed by its FIRST seen chunk; dedupe by ``document.id``), - and at most *n* of them are returned (default the - ``BOR_SUGGESTED_DOCS`` setting, 5). Unlike the phase-113 cited tier, - the usefulness bar NEVER filters here: a lexical-only hit with - cosine 0.0 is suggested when it ranks. Suggestions are opt-in + position is fixed by its FIRST seen chunk; dedupe by ``document.id``) + — the shared :func:`_selection_order` walk — plus, when the bonus is + on AND a name-hit chunk is present, the + ``(−effective, −best_cosine, path, first_seen)`` re-rank that gives + a name-hit document its head start. At most *n* of them are returned + (default the ``BOR_SUGGESTED_DOCS`` setting, 5). Unlike the phase-113 + cited tier, the usefulness bar NEVER filters here: a lexical-only hit + with cosine 0.0 is suggested when it ranks. Suggestions are opt-in starting points, not citations — the seeded prompt carries the document's summary, and the LLM decides whether to extend its context by reading the document's full text. + *bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the + owner-tunable starting point); ``0`` reproduces the pre-phase walk + byte-identically (the kill switch, LOCKED A3). + The returned rows carry the full document content, byte-identical — the content is what the agent's ``read`` tool serves later (never truncated; A6 re-revises A7: full text enters the context only through the capped ``read`` tool). """ top_n = n if n is not None else get_settings().suggested_docs - - order: list[Document] = [] - seen: set[uuid.UUID] = set() - for rc in sorted(chunks, key=lambda c: c.score, reverse=True): - if len(order) >= top_n: - break - doc = rc.document - if doc.id in seen: - continue - seen.add(doc.id) - order.append(doc) - return order + if bonus is None: + bonus = get_settings().name_hit_bonus + return [doc for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)[:top_n]] def select_related( chunks: Sequence[RetrievedChunk], excluded_ids: set[uuid.UUID], cap: int, + bonus: float | None = None, ) -> list[Document]: """The documents ranked AFTER *excluded_ids* — the phase-118 related tier (rank 6+ for the contiguous top-5 suggestion set), up to *cap* (``BOR_RELATED_MAX_DOCS``). - The SAME stable score-descending walk as - :func:`select_documents_tiered` / :func:`select_suggested` (a - document's rank position is fixed by its FIRST seen chunk; dedupe by - ``document.id``), skipping every document whose id is in + The SAME shared selection walk as :func:`select_documents_tiered` / + :func:`select_suggested` (a document's rank position is fixed by its + FIRST seen chunk; dedupe by ``document.id`` — the phase-119 name-hit + bonus applied, D2, LOCKED A3), skipping every document whose id is in *excluded_ids* and admitting at most *cap* documents. There is NO cosine floor: the related tier is the ranked remainder (a lexical- only cosine 0.0 hit is included) — its job on the ``done`` frame is @@ -741,18 +921,21 @@ def select_related( tier's document ids (LOCKED A3: a contiguous top-N, no floor) — "excluding the suggested" is exactly "rank 6+". + *bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the + owner-tunable starting point); ``0`` reproduces the pre-phase walk + byte-identically (the kill switch, LOCKED A3). + The returned rows carry the full document content, byte-identical (the tier is metadata for the ``done`` frame and the durable record; the prompt and ``read`` contract are untouched). """ + if bonus is None: + bonus = get_settings().name_hit_bonus out: list[Document] = [] - seen: set[uuid.UUID] = set() - for rc in sorted(chunks, key=lambda c: c.score, reverse=True): + for doc, _eff, _cos, _idx in _selection_order(chunks, bonus): if len(out) >= cap: break - doc = rc.document - if doc.id in seen or doc.id in excluded_ids: + if doc.id in excluded_ids: continue - seen.add(doc.id) out.append(doc) return out diff --git a/scripts/eval_retrieval.py b/scripts/eval_retrieval.py index 2bcf158..b7285c0 100644 --- a/scripts/eval_retrieval.py +++ b/scripts/eval_retrieval.py @@ -5,7 +5,9 @@ uses (cosine top-N + FTS top-N, RRF-fused — plus the phase-106 recency boost, which ``retrieve()`` applies after the fusion), and prints the top-5 documents with their cosine / fts / fused scores (labelled ``effective`` when the recency boost is on — the post-boost score) -and the document's creation date, plus the honesty-gate verdict: +and the document's creation date, the seeded suggestion tier +(``suggested:`` — the ``select_suggested`` walk, the phase-119 name-hit +bonus included), plus the honesty-gate verdict: uv run python -m scripts.eval_retrieval "How did I install gitlab?" uv run python -m scripts.eval_retrieval --from-file questions.txt @@ -86,7 +88,7 @@ def main(argv: list[str] | None = None) -> int: from app.config import get_settings from app.db import SessionLocal, db_available from app.rag.llm import LLMClient - from app.rag.retriever import RetrievedChunk, retrieve + from app.rag.retriever import RetrievedChunk, retrieve, select_suggested settings = get_settings() if not db_available(): @@ -101,7 +103,8 @@ def main(argv: list[str] | None = None) -> int: f"vector_candidates={settings.hybrid_vector_candidates} " f"lexical_candidates={settings.hybrid_lexical_candidates} rrf_k={settings.rrf_k} " f"recency_boost={settings.recency_boost} " - f"recency_half_life_days={settings.recency_half_life_days}" + f"recency_half_life_days={settings.recency_half_life_days} " + f"name_hit_bonus={settings.name_hit_bonus}" ) with SessionLocal() as db: for question, vec in zip(questions, vectors, strict=True): @@ -134,6 +137,21 @@ def main(argv: list[str] | None = None) -> int: f"{score_label}={c.score:.5f} created={c.document.created_at:%Y-%m-%d} " f"({c.document.title})" ) + # Phase 119, D2: the seeded suggestion tier — the + # ``select_suggested`` walk over the same list (the name-hit + # bonus included, default settings) — the tuning tool must + # report the tier the prompt actually seeds. + suggested = select_suggested(chunks) + print( + " suggested: " + + ( + " ".join( + f"{i}. {d.source}/{d.path}" + for i, d in enumerate(suggested, start=1) + ) + or "(none)" + ) + ) return 0 diff --git a/tests/e2e/mock_llm.py b/tests/e2e/mock_llm.py index be89917..81320b5 100644 --- a/tests/e2e/mock_llm.py +++ b/tests/e2e/mock_llm.py @@ -436,6 +436,29 @@ Implements just enough of the aipi surface: phrases — the phase-71/72/94 ordering convention); verified 2026-09-16: no existing E2E question or fixture file contains the phrase, so every other suite is unaffected. + - user message containing ``repeat your folder map`` + (``FOLDER_MAP_TRIGGER``, phase 119, D3 — the suggested-folder + context's story suite ``tests/e2e/test_name_signal_read_chips.py``) + **and** the system prompt contains the ```` section -> + the composed answer ends with `` (folders: ; ; …)`` + echoing the suggested-folder lines VERBATIM — the plain lines + between ``SUGGEST_INTRO`` and the first ```` section (``app.rag.prompts.build_high_prompt`` / + ``app.rag.agent.suggested_folder_lines``, phase 119 LOCKED A4: + ``//: e1, e2, …``), joined with ``; ``. If no + folder lines are present (empty *folder_lines* — the + byte-identical phase-118 shape) the echo is omitted and the answer + is otherwise unchanged. The same prompt-injection-echo convention + as the ```` / ```` triggers above (a SUFFIX + appended after both, so the folders suffix is the last thing + rendered): phase 119 D3's observability — the folder lines land in + the model's prompt only on grounded turns, and the echo is the + E2E's deterministic lens on their EXACT text (line identity shape + + entry order + the owning-doc exclusion + the caps). It composes + with any base answer (it does not compete with a marker branch); + the trigger phrase is disjoint from every other trigger's; verified + 2026-09-16: no existing E2E question or fixture file contains the + phrase, so every other suite is unaffected. Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic dead-endpoint behavior for the retry E2E suite (``tests/e2e/ @@ -675,6 +698,17 @@ TABLE_TRIGGER = "show me a table" #: phrase, so every other suite is unaffected. HISTORY_TRIGGER = "echo my history" +#: Phase 119 (D3, the suggested-folder context's story suite): a user +#: message containing this substring (case-insensitive) — combined +#: with the ```` section in the system prompt (a grounded +#: turn) — gets the composed answer suffixed with the folder lines +#: VERBATIM (`` (folders: ; ; …)``, joined with +#: ``; ``; omitted when the section carries no folder lines) — see the +#: module docstring. Disjoint from every other trigger phrase; verified +#: 2026-09-16: no existing E2E question or fixture file contains the +#: phrase, so every other suite is unaffected. +FOLDER_MAP_TRIGGER = "repeat your folder map" + TABLE_ANSWER = ( "Here's the shape, in a table:\n" "\n" @@ -1657,6 +1691,32 @@ def first_kb_bullet(system: str) -> str | None: return None +def folder_map_lines(system: str) -> list[str]: + """The suggested-folder lines of the ```` section + (phase 119, D3, LOCKED A4) — or ``[]`` when there are none. + + ``build_high_prompt`` lays the section body out as ``SUGGEST_INTRO`` + (line 1), the folder lines (each on its own line, immediately after + the intro line), a blank line, then the first ```` block + — so the lines are exactly the non-blank lines between the intro + line and the first `` str: """The phase-74 history echo (byte-stable, stateless over messages). @@ -2093,6 +2153,18 @@ def compose_answer(body: dict[str, Any]) -> str: bullet = first_kb_bullet(system) if bullet: answer = f"{answer} (kb: {bullet})" + # Suggested-folder context (phase 119, D3, LOCKED A4): when the user + # message carries FOLDER_MAP_TRIGGER and the system prompt carries + # the section, the answer ends with the folder lines + # VERBATIM (joined with "; ") — the E2E's deterministic lens on the + # exact lines build_high_prompt seeded (D3 observability; the same + # prompt-injection-echo convention as the two echoes above). Omitted + # when the section carries no folder lines (the byte-identical + # phase-118 shape) — the answer is otherwise unchanged. + if FOLDER_MAP_TRIGGER in user.lower() and "" in system: + folder_lines = folder_map_lines(system) + if folder_lines: + answer = f"{answer} (folders: {'; '.join(folder_lines)})" return answer diff --git a/tests/e2e/test_agent_document_tools.py b/tests/e2e/test_agent_document_tools.py index 428d8f3..33d1fa0 100644 --- a/tests/e2e/test_agent_document_tools.py +++ b/tests/e2e/test_agent_document_tools.py @@ -60,12 +60,15 @@ Test → story mapping (Playwright Mapping Rule): ``tool`` frames (``ls``, then ``read`` with the combined path, ahead of any delta), the UI shows the transient calling-tool status while a tool runs, the bubble shows both tool lines, the final answer quotes - the read document, and the source chips include the read document - (viewer link). + the read document, and the source chips are EXACTLY the read + document (phase 119, LOCKED A1 — chips cite read docs only; viewer + link). 2. ``test_tool_lines_re_render_after_reload`` — the persisted record (phase 14) re-renders the tool lines. 3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no - ``tool`` frames, the answer renders exactly as today (regression + ``tool`` frames, the answer renders exactly as today, and (phase 119, + LOCKED A1) the zero-read grounded turn chips NOTHING (the retrieval + doc is cited only in the durable record, never as a chip) (regression inside the story file). 4. ``test_deflected_question_has_no_tool_frames`` — the tools are grounded-only: a deflected turn runs none. @@ -444,8 +447,12 @@ def test_marker_question_lists_reads_and_quotes( ) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False + # Phase 119 (LOCKED A1): done.sources = the READ docs only — + # exactly the scripted read; the retrieval (seed) doc is suggested + # context, never a citation chip (the retired phase-118 A4 + # suggested+read union is gone). The durable record below still + # carries both (LOCKED A3, untouched). assert [(s["source"], s["path"]) for s in done["sources"]] == [ - (SEED_SOURCE, SEED_PATH), (READ_SOURCE, READ_PATH), ] @@ -465,16 +472,17 @@ def test_marker_question_lists_reads_and_quotes( expect(bubble).to_contain_text(ANSWER_PREFIX) expect(bubble).to_contain_text(ANSWER_QUOTE) - # Source chips: the retrieval doc AND the read doc (deduped, in - # order) — the read chip links to the viewer. + # Source chips: the read doc ONLY (phase 119, LOCKED A1 — the + # retrieval doc was never read, so it never chips; the retired + # phase-118 A4 union is gone) — the read chip links to the viewer. chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(2) - expect(chips.nth(0)).to_contain_text(SEED_SP) + expect(chips).to_have_count(1) chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH) expect(chip_read).to_have_count(1) expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF) - # Durable record: grounded, both sources logged (retrieval + read). + # Durable record: grounded, both sources logged (suggested + read — + # LOCKED A3, untouched by phase 119 A1). row = _last_query_log() assert row.question == MARKER_QUESTION assert row.deflected is False @@ -540,14 +548,15 @@ def test_plain_grounded_question_has_no_tool_frames( assert _tool_frames(_frames(page)) == [] expect(page.locator(".tool-call")).to_have_count(0) - # The standard grounded answer, citing the retrieval doc only — the - # referenced JSON stays OUT of the sources (it was never read). + # The standard grounded answer, with ZERO citation chips — the + # turn read nothing, so (phase 119, LOCKED A1) the chip row is + # empty: the retrieval doc is cited in the durable record only + # (the retired phase-118 A4 union is gone); the referenced JSON + # stays out of the record too (it was never read). bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text(PLAIN_QUESTION) expect(bubble).to_contain_text(MOCK_ANSWER_MARKER) - chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(1) - expect(chips.first).to_contain_text(SEED_SP) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) row = _last_query_log() assert row.question == PLAIN_QUESTION diff --git a/tests/e2e/test_agent_unlimited_tools.py b/tests/e2e/test_agent_unlimited_tools.py index cf62b27..ad38970 100644 --- a/tests/e2e/test_agent_unlimited_tools.py +++ b/tests/e2e/test_agent_unlimited_tools.py @@ -65,9 +65,11 @@ Test → story mapping (Playwright Mapping Rule): ``I read and .`` line; the round cap (default 10) bounds the turn, no budget refusal anywhere. 2. ``test_done_sources_include_reads`` — the source chips under the - answer list the retrieval doc PLUS both read documents, deduped - (the phase-37 ``done.sources`` extension contract, now with 2 - reads); the same combined list lands in ``query_log.sources``. + answer list the BOTH read documents, deduped (phase 119, LOCKED A1: + chips cite read docs only — the retrieval doc was never read, so it + never chips; the retired phase-118 A4 suggested+read union is + gone); the retrieval doc PLUS both reads still land in + ``query_log.sources`` (LOCKED A3, untouched). 3. ``test_relist_allowed`` — the listing tool ran (its line rendered) and no pre-phase-45 budget refusal ("… budget left") appears anywhere in the message bubble or tool lines: the old @@ -195,14 +197,22 @@ BUDGET_REFUSAL_FRAGMENTS = ( "budget left", ) -# The combined source list the app reports (app/api/chat.py): retrieval -# docs first, then the agent's read docs, deduped by (source, path). +# The combined source list the DURABLE record reports +# (app/api/chat.py, LOCKED A3 — suggested + related + read, deduped): +# retrieval doc first, then the agent's read docs, deduped by +# (source, path). The citation surface (done.sources, the chips) is +# the READ set only (phase 119, LOCKED A1 — the retired phase-118 A4 +# suggested+read union is gone): the retrieval doc was never read. EXPECTED_SOURCES = [ (SEED_SOURCE, SEED_PATH), (READ1_SOURCE, READ1_PATH), (READ2_SOURCE, READ2_PATH), ] EXPECTED_SOURCES_LINE = ", ".join(f"{s}/{p}" for s, p in EXPECTED_SOURCES) +READ_ONLY_SOURCES = [ + (READ1_SOURCE, READ1_PATH), + (READ2_SOURCE, READ2_PATH), +] # -------------------------------------------------------------------------- @@ -508,7 +518,7 @@ def test_multi_read_turn( # -------------------------------------------------------------------------- -# 2. done.sources / source chips: retrieval doc + BOTH reads, deduped +# 2. done.sources / source chips: BOTH reads, deduped (phase 119 A1) # -------------------------------------------------------------------------- @@ -523,23 +533,28 @@ def test_done_sources_include_reads( _submit(page, MULTI_QUESTION) _wait_settled(page) - # Wire level: done.sources is the retrieval doc FIRST, then both - # read documents — deduped (the retrieval doc was never read, the - # reads are each read once; nothing appears twice). + # Wire level: done.sources is the READ documents only, in read + # order — deduped (the reads are each read once; nothing appears + # twice). Phase 119, LOCKED A1: the retrieval doc was never read, + # so it is NOT in the citation surface (the retired phase-118 A4 + # suggested+read union is gone); it still lands in the durable + # record (LOCKED A3, pinned in test 1). frames = _frames(page) done = next(f for f in frames if f.get("type") == "done") - assert [(s["source"], s["path"]) for s in done["sources"]] == EXPECTED_SOURCES + assert [ + (s["source"], s["path"]) for s in done["sources"] + ] == READ_ONLY_SOURCES, done["sources"] pairs = [(s["source"], s["path"]) for s in done["sources"]] assert len(pairs) == len(set(pairs)), "done.sources must be deduped" - # UI: exactly three source chips under the answer, in the same - # order, each a viewer link — no duplicated chip. + # UI: exactly two source chips under the answer (the read docs, + # read order), each a viewer link — no duplicated chip, and no + # chip for the never-read retrieval doc (phase 119 A1). chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(3) - expect(chips.nth(0)).to_contain_text(SEED_SP) - expect(chips.nth(1)).to_contain_text(READ1_SP) - expect(chips.nth(2)).to_contain_text(READ2_SP) - for i, (source, path) in enumerate(EXPECTED_SOURCES): + expect(chips).to_have_count(2) + expect(chips.nth(0)).to_contain_text(READ1_SP) + expect(chips.nth(1)).to_contain_text(READ2_SP) + for i, (source, path) in enumerate(READ_ONLY_SOURCES): expect(chips.nth(i)).to_have_attribute( "href", f"/document.html?source={source}&path={path}&back=%2F" ) @@ -631,12 +646,13 @@ def test_single_tool_flow_regression( expect(bubble).not_to_contain_text(BOTH_READS_LINE) expect(bubble).not_to_contain_text(READ2_SP) - # done: non-deflected; sources = retrieval doc + the single read - # (READ2 absent — it was never read). + # done: non-deflected; sources = the single READ doc only (phase + # 119, LOCKED A1 — READ2 absent: never read; the retrieval doc + # absent: never read — the retired phase-118 A4 union is gone; the + # durable record below keeps retrieval + read, LOCKED A3). done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False assert [(s["source"], s["path"]) for s in done["sources"]] == [ - (SEED_SOURCE, SEED_PATH), (READ1_SOURCE, READ1_PATH), ] diff --git a/tests/e2e/test_api_tokens.py b/tests/e2e/test_api_tokens.py index c570f84..2c84a01 100644 --- a/tests/e2e/test_api_tokens.py +++ b/tests/e2e/test_api_tokens.py @@ -391,8 +391,11 @@ def test_admin_generates_token_in_ui(page: Page, app_url: str, db_ready: None) - # --------------------------------------------------------------------------- # 4. The token flow: a fresh context signs in through the real gate -# and USES the app — a grounded chat turn (mock LLM), a cited -# document opened in the same-page modal, the role-user header +# and USES the app — a grounded chat turn (mock LLM; phase 119, +# LOCKED A1: the turn is the scripted summary-read flow — chips +# cite READ docs only, and this turn's read earns the cited chip), +# a cited document opened in the same-page modal, the role-user +# header # --------------------------------------------------------------------------- @@ -417,15 +420,27 @@ def test_token_user_uses_the_app( # task-04 helper: fill #auth-gate-input → submit → gate hides). login_with_token(user, app_url, token) - # Use the app: a grounded turn against the seeded KB (mock - # LLM) — the brain bubble renders the deterministic answer. - _ask(user, "How is my Kubernetes cluster set up? (api-tokens-flow)") + # Use the app: a GROUNDED turn against the seeded KB (mock + # LLM). Phase 119 (LOCKED A1): chips cite READ docs only — a + # plain question would chip nothing (the retired phase-118 A4 + # union is gone), so the turn is the mock's scripted + # summary-read flow (SUMMARY_SEED_READ_TRIGGER): it reads the + # kubernetes fixture, and that read is the chip's source. + q = ( + "Read the suggested document: read docs/homelab/kubernetes.md — " + "how is my Kubernetes cluster set up? (api-tokens-flow)" + ) + user.fill("#message-input", q) + user.click("#send-btn") + expect(user.locator(".msg.user .bubble").last).to_contain_text(q) - # A cited source chip opens the document in the SAME-PAGE - # modal (the require_user content endpoint passes for a - # live token session). + # The turn's cited source chip (its one READ doc — the scripted + # read, phase 119 A1) opens the document in the SAME-PAGE modal + # (the require_user content endpoint passes for a live token + # session). chip = user.locator(".msg.brain a.source-chip").first - expect(chip).to_be_visible(timeout=15_000) + expect(chip).to_be_visible(timeout=30_000) + expect(user.locator("#send-label")).to_have_text("Send") chip.click() expect(user.locator("#doc-modal")).to_be_visible() expect( diff --git a/tests/e2e/test_chat_history.py b/tests/e2e/test_chat_history.py index a225675..e2b2346 100644 --- a/tests/e2e/test_chat_history.py +++ b/tests/e2e/test_chat_history.py @@ -63,9 +63,6 @@ from e2e.auth_helpers import login REPO = Path(__file__).resolve().parents[2] FIXTURES = REPO / "tests" / "fixtures" / "docs" MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E" -#: Phase 10 viewer URL + phase 13 back=/ (the restored chip must be -#: byte-identical to the live-rendered one). -CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F" async def _import_fixtures(mock_port: int) -> ImportSummary: @@ -279,10 +276,9 @@ def test_open_chat_returns_to_history( # …the SAME answer text the History session saw (pixel-identical # restore through renderStoredMessage)… assert bubble.inner_text() == answer_before - # …with its source chip restored byte-identically. - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip).to_have_count(1) - expect(chip.first).to_have_attribute("href", CHIP_HREF) + # …with ZERO citation chips restored — the saved turn read + # nothing, so its sources list is empty (phase 119, LOCKED A1; + # the retired phase-118 A4 suggested-chip pin is gone). # The conversation continues: a new turn streams fine… _ask(page, "How is my Kubernetes cluster set up? (hist-open-2)") diff --git a/tests/e2e/test_chat_persistence.py b/tests/e2e/test_chat_persistence.py index 2290aab..1784019 100644 --- a/tests/e2e/test_chat_persistence.py +++ b/tests/e2e/test_chat_persistence.py @@ -42,9 +42,6 @@ OFF_TOPIC = "How do I bake sourdough bread?" MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E" DEFLECT_PHRASE = r"haven't done anything like that" STORAGE_KEY = "bor.chat.v1" -#: Phase 10 viewer URL + phase 13 back=/ (the restored chip must be -#: byte-identical to the live-rendered one). -CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F" async def _import_fixtures(mock_port: int) -> ImportSummary: @@ -143,22 +140,24 @@ def test_conversation_survives_reload( assert MOCK_ANSWER_MARKER in brain["text"] assert "<" not in brain["text"], "persisted brain text must be raw, not rendered HTML" assert brain["deflected"] is False - assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"]) + # Phase 119 (LOCKED A1): the turn read nothing, so its done sources + # — and the persisted record — are EMPTY (the suggested kubernetes + # doc is context, not a citation; the retired phase-118 A4 union is + # gone). # Refresh — the same context keeps its localStorage. page.reload() expect(page.locator("#empty-state")).to_be_hidden() - # Both bubbles restored: text + the source chip with the exact viewer URL. + # Both bubbles restored: the answer text, and ZERO citation chips + # (the zero-read turn persisted an empty sources list — phase 119, + # LOCKED A1; the retired phase-118 A4 chip pin is gone). expect(page.locator(".msg.user .bubble")).to_have_count(1) expect(page.locator(".msg.user .bubble")).to_contain_text(QUESTION) bubble = page.locator(".msg.brain .bubble") expect(bubble).to_have_count(1) expect(bubble).to_contain_text(MOCK_ANSWER_MARKER) - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip).to_have_count(1) - expect(chip.first).to_have_attribute("href", CHIP_HREF) - expect(chip.first).not_to_have_attribute("target") # phase 26: modal, not a new tab + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # The restore is read-only: storage still holds the same two messages. assert [m["who"] for m in _stored_parsed(page)["messages"]] == ["user", "brain"] @@ -294,7 +293,9 @@ def test_persists_across_page_navigation( expect(page.locator(".msg.user .bubble").first).to_contain_text(QUESTION) expect(page.locator(".msg.user .bubble").nth(1)).to_contain_text(OFF_TOPIC) expect(page.locator(".msg.brain .bubble").first).to_contain_text(MOCK_ANSWER_MARKER) - expect(page.locator(".msg.brain .source-chip", has_text="kubernetes.md")).to_have_count(1) + # Zero citation chips (phase 119, LOCKED A1 — the grounded turn read + # nothing; the retired phase-118 A4 union is gone). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) deflected = page.locator(".msg.brain.is-deflected .bubble") expect(deflected).to_have_count(1) expect(deflected.first).to_contain_text(re.compile(DEFLECT_PHRASE, re.IGNORECASE)) diff --git a/tests/e2e/test_chat_rag.py b/tests/e2e/test_chat_rag.py index 3328719..abe3cbf 100644 --- a/tests/e2e/test_chat_rag.py +++ b/tests/e2e/test_chat_rag.py @@ -97,18 +97,12 @@ def test_on_topic_question_streams_grounded_answer( expect(bubble.first).to_contain_text(QUESTION, timeout=30_000) expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) - # Grounded: a kubernetes.md source chip renders under the bubble - # (top-N docs can add more chips; the question's doc must be among them). - # Phase 26: the chip opens the document in the SAME-PAGE modal — no new - # tab; the encoded href stays as the no-JS / context-menu escape hatch - # (phase 13's back=/ lets the viewer's back button return to chat). - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip).to_have_count(1) - expect(chip.first).to_contain_text("kubernetes.md") - expect(chip.first).to_have_attribute( - "href", "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F" - ) - expect(chip.first).not_to_have_attribute("target") # phase 26: modal, not a new tab + # Phase 119 (LOCKED A1): a zero-read grounded turn chips NOTHING — + # the suggested kubernetes.md is seed context, not a citation chip + # (the retired phase-118 A4 suggested+read union is gone; the + # pre-phase-26 chip/contract pins retired with it). The grounding is + # pinned by the query_log row below (LOCKED A3, untouched). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # Button recovers: enabled + "Send" (never stale). expect(page.locator("#send-btn")).to_be_enabled() @@ -121,9 +115,16 @@ def test_chat_logs_query(page: Page, app_url: str, mock_llm: int, db_ready: None login(page, app_url, next="/") page.fill("#message-input", QUESTION) page.click("#send-btn") - expect( - page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - ).to_have_count(1, timeout=30_000) + # The turn settles with the mock answer (phase 119 A1: no chip to + # wait on — a zero-read turn chips nothing; the retired phase-118 + # A4 union is gone). The button recovery is the settle sync: the + # client re-enables Send on the done frame, and the server writes + # the query_log row just before yielding it. + expect(page.locator(".msg.brain .bubble").last).to_contain_text( + "Deterministic mock answer for E2E", timeout=30_000 + ) + expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000) + expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) # App still healthy after the turn. r = httpx.get(f"{app_url}/api/health", timeout=5) @@ -180,5 +181,8 @@ def test_sse_stream_shape(app_url: str, mock_llm: int, db_ready: None) -> None: assert frames[-1]["type"] == "done" # done is the final event assert done[0]["deflected"] is False assert done[0]["suggestions"] == [] - assert done[0]["sources"], "done must carry the cited sources" - assert any(s["path"] == "homelab/kubernetes.md" for s in done[0]["sources"]) + # Phase 119 (LOCKED A1): done.sources = the READ docs only — this + # plain turn read nothing, so the frame carries an EMPTY sources + # list (the retired phase-118 A4 suggested+read union is gone; the + # suggested doc's durable record lives in query_log, not the frame). + assert done[0]["sources"] == [], done[0]["sources"] diff --git a/tests/e2e/test_dark_tech_theme.py b/tests/e2e/test_dark_tech_theme.py index 0befb6a..7adbf5d 100644 --- a/tests/e2e/test_dark_tech_theme.py +++ b/tests/e2e/test_dark_tech_theme.py @@ -358,8 +358,9 @@ def test_behavior_unchanged_smoke( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: """AC6: layout/behavior unchanged under the new skin — an on-topic - question streams a grounded answer, renders a source chip, and the send - button recovers (never stale).""" + question streams a grounded answer (phase 119, LOCKED A1: a zero-read + turn renders ZERO source chips — the retired phase-118 A4 + suggested-chip is gone) and the send button recovers (never stale).""" _seed_kb(mock_llm) page.set_default_timeout(30_000) login(page, app_url, next="/") # phase 79: chat is require_user-gated @@ -373,7 +374,8 @@ def test_behavior_unchanged_smoke( bubble.first.wait_for(state="visible", timeout=30_000) expect(bubble.first).to_contain_text(QUESTION, timeout=30_000) expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) - expect(page.locator(".msg.brain .source-chip", has_text="kubernetes.md")).to_have_count(1) + # Phase 119 (LOCKED A1): zero-read turn → zero citation chips. + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # The state machine settled: button re-enabled, label back to "Send". expect(page.locator("#send-btn")).to_be_enabled() diff --git a/tests/e2e/test_document_back_navigation.py b/tests/e2e/test_document_back_navigation.py index 77d558b..f83f63b 100644 --- a/tests/e2e/test_document_back_navigation.py +++ b/tests/e2e/test_document_back_navigation.py @@ -19,10 +19,16 @@ viewer URL survives as each link's ``href`` (the no-JS / context-menu "open in new tab" escape hatch), so the back contract is asserted on that exact href and verified by navigating to it directly. +Phase 119 re-target (LOCKED A1): chips cite READ docs only — a plain +question chips nothing, so test 1 drives the mock's scripted +summary-read flow (``SUMMARY_SEED_READ_TRIGGER``): the turn ``read``s +the kubernetes fixture, and its READ-doc chip is what carries the +``&back=%2F`` href the story asserts. + Test → story mapping (Playwright Mapping Rule): -1. ``test_back_from_chat_returns_to_chat`` — question → source chip href - (carries ``&back=%2F``) → viewer back link href ``/`` labeled "Chat" - → click → the chat page. +1. ``test_back_from_chat_returns_to_chat`` — scripted-read question → + the read doc's source chip href (carries ``&back=%2F``) → viewer + back link href ``/`` labeled "Chat" → click → the chat page. 2. ``test_back_from_sources_returns_to_sources`` — Sources table link href (no ``back`` param) → back link href ``/sources.html`` labeled "Sources" → click → the Sources page. @@ -49,7 +55,15 @@ from e2e.auth_helpers import login REPO = Path(__file__).resolve().parents[2] FIXTURES = REPO / "tests" / "fixtures" / "docs" -QUESTION = "How is my Kubernetes cluster set up?" +# Phase 119 (LOCKED A1): the chip is the turn's READ doc — the scripted +# summary-read flow (mock_llm.SUMMARY_SEED_READ_TRIGGER) reads the +# kubernetes fixture so the story's chip (with its back=%2F href) +# exists (a zero-read turn would chip nothing — the retired phase-118 +# A4 suggested-chip is gone). +QUESTION = ( + "Read the suggested document: read docs/homelab/kubernetes.md — " + "how is my Kubernetes cluster set up?" +) # Seeded fixture doc (source=docs) shared by every test in this file. DOC_SOURCE = "docs" DOC_PATH = "homelab%2Fkubernetes.md" @@ -111,6 +125,8 @@ def test_back_from_chat_returns_to_chat( chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") expect(chip).to_have_count(1, timeout=30_000) + # Phase 119 A1: the read doc is the turn's ONLY chip. + expect(page.locator(".msg.brain .source-chip")).to_have_count(1) # Chat chips carry back=/ (encoded %2F) so the viewer knows where # home is. Phase 26: the left click opens the same-page modal (no # target=_blank); this href is what the no-JS / context-menu "open diff --git a/tests/e2e/test_document_dates.py b/tests/e2e/test_document_dates.py index 0e48208..7f18ecc 100644 --- a/tests/e2e/test_document_dates.py +++ b/tests/e2e/test_document_dates.py @@ -540,12 +540,14 @@ def test_old_correct_beats_new_similar( page: Page, app_url: str, mock_llm: int, db_ready: None, dates_tree: Path ) -> None: """``How did I configure the backup retention policy?`` → grounded, - and the FIRST cited source is the OLDER correct doc (2020) — the - NEWER similar one (now, "under review") is cited second. Phase 118 - (A4): the chip row IS the suggested tier (top-5, NO floor) — this - four-doc KB chips ALL four docs in rank order — and the related row - is absent (no rank-6+ doc). The real hybrid retriever + the DEFAULT - recency boost (0.0007 / 365 d) over the mock's token-overlap + and the FIRST logged source is the OLDER correct doc (2020) — the + NEWER similar one (now, "under review") is second. Phase 119 + (LOCKED A1): the zero-read turn chips NOTHING (the chip row is the + READ docs only — the retired phase-118 A4 suggested-chip row is + gone), so the ordering assertion rides the durable record (LOCKED + A3 — all four docs, suggested rank order, untouched); the related + row is absent (no rank-6+ doc). The real hybrid retriever + the + DEFAULT recency boost (0.0007 / 365 d) over the mock's token-overlap embeddings (the module docstring records the measured fused scores: 0.032523 vs 0.031498 — margin ≈ 0.001025 WITH the full zero-age boost on the newer doc).""" @@ -563,21 +565,20 @@ def test_old_correct_beats_new_similar( # top_score 0.6222 ≥ the e2e threshold 0.30). expect(page.locator(".msg.brain.is-deflected")).to_have_count(0) - # Phase 118 (A4): the chip row IS the suggested tier — top-5, NO - # floor, and this KB has exactly four docs, so ALL four are - # suggested (chipped) in fused rank order: the OLDER correct doc - # first, the NEWER similar one (the boost's intended beneficiary) - # second, then the two unrelated docs (no floor filters them — the - # LLM decides what the summaries earn). - chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(4, timeout=30_000) - assert chips.nth(0).inner_text() == f"{dates_tree.name}/{RETENTION_MD}" - assert chips.nth(1).inner_text() == f"{dates_tree.name}/{DRAFT_MD}" - assert chips.nth(2).inner_text() == f"{dates_tree.name}/{FORWARD_MD}" - assert chips.nth(3).inner_text() == f"{dates_tree.name}/{OLDDOC_MD}" + # Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation + # chips — the four suggested docs (all of them, top-5 NO floor on a + # four-doc KB: the OLDER correct doc first, the NEWER similar one + # second, then the two unrelated docs) seeded the prompt but never + # chip (the retired phase-118 A4 union is gone). Their rank order + # is pinned by the durable record below (LOCKED A3, untouched). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # No rank-6+ doc exists in this four-doc KB → the related row is # absent (the de-emphasized row renders only when it has entries). expect(page.locator(".msg.brain .related-docs")).to_have_count(0) + # The button label is the settle sync: the client re-labels Send on + # the done frame, and the server writes the query_log row just + # before yielding it. + expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) # Durable record: one row, grounded, the FULL retrieval (suggested # tier + related + read, deduped — here: all four docs) in rank diff --git a/tests/e2e/test_document_summaries.py b/tests/e2e/test_document_summaries.py index b15bb08..f6b9ca4 100644 --- a/tests/e2e/test_document_summaries.py +++ b/tests/e2e/test_document_summaries.py @@ -171,8 +171,9 @@ def test_summary_hit_seeds_the_summary_not_the_full_text( quotes the last suggested doc's summary tail (digest + pointer), and the sentinel (the yaml doc's last line, outside the digest) is ABSENT: the full source doc never reached the prompt (A6). Both - fixture docs are suggested (two-doc KB, no floor) and both chips - render (deflected: false).""" + fixture docs are suggested (two-doc KB, no floor) but chip + NOTHING — the turn read nothing, so (phase 119, LOCKED A1) the + citation surface is empty (deflected: false).""" _reset_db() summary = _run_in_thread(_import_fixtures(mock_llm)) assert summary.added == 2 # yaml + md control @@ -225,13 +226,13 @@ def test_summary_hit_seeds_the_summary_not_the_full_text( expect(bubble).to_contain_text(yaml_digest_line[-80:]) expect(bubble).not_to_contain_text(SENTINEL) - # Grounded: both fixture docs are suggested (two-doc KB — no floor) - # and both chips render, in rank order (md first, yaml last). - chip = page.locator(".msg.brain .source-chip", has_text=YAML_PATH) - expect(chip).to_have_count(1) - expect(chip.first).to_contain_text(f"{SOURCE}/{YAML_PATH}") - expect(page.locator(".msg.brain .source-chip", has_text=MD_PATH)).to_have_count(1) - expect(page.locator(".msg.brain .source-chip")).to_have_count(2) + # Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation + # chips — both fixture docs are suggested (two-doc KB — no floor, + # md first, yaml last) but suggested docs are seed context, not + # citations (the retired phase-118 A4 suggested-chip union is gone); + # their rank order is pinned by the durable record below (LOCKED + # A3, untouched). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # No rank-6+ doc in a two-doc KB → the related row is absent. expect(page.locator(".msg.brain .related-docs")).to_have_count(0) diff --git a/tests/e2e/test_document_viewer.py b/tests/e2e/test_document_viewer.py index 9d94b7b..0aa4e84 100644 --- a/tests/e2e/test_document_viewer.py +++ b/tests/e2e/test_document_viewer.py @@ -11,9 +11,18 @@ deterministic mock embeddings (same harness as the phase-10 suite — only the assertions changed: chips/row links now open the SAME-PAGE modal, no ``expect_popup``). +Phase 119 re-target (LOCKED A1): chips cite READ docs only — a plain +question chips nothing, so the chat-driven tests here run the +mock's scripted summary-read flow (``SUMMARY_SEED_READ_TRIGGER`` — +the house scripted-turn convention): the turn ``read``s +``docs/homelab/kubernetes.md``, whose read result is what earns the +kubernetes.md chip the story interacts with (the chip's viewer-link +contract is unchanged). + Test → story mapping (Playwright Mapping Rule): -1. ``test_source_chip_opens_modal`` — chat chip → modal opens in-page - (NO new tab, URL unchanged), title + ``.doc-md`` content + meta row. +1. ``test_source_chip_opens_modal`` — chat chip (the turn's READ doc — + phase 119 A1) → modal opens in-page (NO new tab, URL unchanged), + title + ``.doc-md`` content + meta row. 2. ``test_sources_row_opens_modal`` — Sources path link → modal, yaml in ````, mono font, URL unchanged. 3. ``test_modal_closes_on_button_escape_and_backdrop`` — close via @@ -56,7 +65,14 @@ from e2e.auth_helpers import login REPO = Path(__file__).resolve().parents[2] FIXTURES = REPO / "tests" / "fixtures" / "docs" SCREENSHOTS = REPO / ".agents" / "screenshots" # house convention for visual records -QUESTION = "How is my Kubernetes cluster set up?" +# Phase 119 (LOCKED A1): the chip is the turn's READ doc — the scripted +# summary-read flow (mock_llm.SUMMARY_SEED_READ_TRIGGER) reads the +# kubernetes fixture so the story's chip exists (the retired phase-118 +# A4 suggested-chip is gone: a zero-read turn would chip nothing). +QUESTION = ( + "Read the suggested document: read docs/homelab/kubernetes.md — " + "how is my Kubernetes cluster set up?" +) async def _import_fixtures(mock_port: int) -> ImportSummary: @@ -98,12 +114,20 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None: def _ask_for_chip(page: Page, app_url: str) -> Any: - """Drive one chat turn and return the kubernetes.md source chip.""" + """Drive one chat turn and return the kubernetes.md source chip. + + The scripted summary-read flow (phase 119, LOCKED A1): the turn + ``read``s the kubernetes doc, so it is the turn's ONLY citation + chip (chips cite read docs only — the answer is the mock's + verbatim echo of the read result).""" login(page, app_url, next="/") # phase 79: chat is require_user-gated page.fill("#message-input", QUESTION) page.click("#send-btn") chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") expect(chip).to_have_count(1, timeout=30_000) + # Phase 119 A1: the read doc is the turn's ONLY chip (exactly one + # .source-chip under the bubble — nothing suggested-but-unread). + expect(page.locator(".msg.brain .source-chip")).to_have_count(1) return chip diff --git a/tests/e2e/test_harness_aligned_tools.py b/tests/e2e/test_harness_aligned_tools.py index f5a701c..8b1aac9 100644 --- a/tests/e2e/test_harness_aligned_tools.py +++ b/tests/e2e/test_harness_aligned_tools.py @@ -455,10 +455,10 @@ def test_read_flow_lines_answer_sources_no_raw_markup( ) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False - # Done-state sources include the read document (retrieval doc first, - # the agent's read doc after — the phase-37 extension contract). + # Phase 119 (LOCKED A1): done.sources = the READ docs only — + # exactly the scripted read; the retrieval (seed) doc was never + # read, so it never cites (the retired phase-118 A4 union is gone). assert [(s["source"], s["path"]) for s in done["sources"]] == [ - (SEED_SOURCE, SEED_PATH), (READ_SOURCE, READ_PATH), ] diff --git a/tests/e2e/test_llm_history.py b/tests/e2e/test_llm_history.py index 87deb5d..98c2032 100644 --- a/tests/e2e/test_llm_history.py +++ b/tests/e2e/test_llm_history.py @@ -23,10 +23,12 @@ assistant message — A4). The marker is checked BEFORE the mock's ``DEFLECT_MODE`` branch, so the echo fires on BOTH turn branches — the branch under test is -discriminated separately (the grounded source chip / the -``is-deflected`` bubble class). The echo answers carry no tool -markup, so no marker tool flow is re-triggered by the now-always- -present (user/assistant-only) history. +discriminated separately (the persisted record's ``deflected`` flag / +the ``is-deflected`` bubble class — phase 119, LOCKED A1: a zero-read +grounded turn chips nothing, so the retired chip discriminator is +gone). The echo answers carry no tool markup, so no marker tool flow +is re-triggered by the now-always-present (user/assistant-only) +history. The file name deliberately differs from phase 50's ``test_chat_history.py`` (save & view chat history — a different @@ -170,12 +172,15 @@ def test_followup_receives_history_and_thinking( expect(bubble).to_contain_text(f"last answer tail: {answer_tail}") expect(bubble).to_contain_text("thinking: yes") # Grounded proof — the echo fires in BOTH branches, so the branch - # is discriminated by the kubernetes.md source chip (the deflected - # turn carries no cited sources). Scoped to the LAST brain message: - # turn 1 cited kubernetes.md too. + # is discriminated by the persisted record's ``deflected`` flag + # (phase 119, LOCKED A1: the turn read nothing, so it also chips + # nothing — the retired chip discriminator is gone). + brain2 = _wait_record(page, 4)["messages"][3] + assert brain2["who"] == "brain" + assert brain2["deflected"] is False, "the echo turn must be the grounded branch" expect( - page.locator(".msg.brain").last.locator(".source-chip", has_text="kubernetes.md") - ).to_have_count(1) + page.locator(".msg.brain").last.locator(".source-chip") + ).to_have_count(0) def test_first_question_has_no_history( @@ -194,11 +199,15 @@ def test_first_question_has_no_history( expect(bubble).to_contain_text("history: 0 prior messages", timeout=30_000) expect(bubble).to_contain_text("last answer tail: none") expect(bubble).to_contain_text("thinking: no") - # Grounded: the echo question is on-topic (the chip proves the - # HIGH gate, not a deflection). + # Grounded: the echo question is on-topic (the record's deflected + # flag proves the HIGH gate, not a deflection — phase 119, LOCKED + # A1: the zero-read turn chips nothing, so the retired chip + # discriminator is gone). + record = _wait_record(page, 2) + assert record["messages"][1]["deflected"] is False expect( - page.locator(".msg.brain").last.locator(".source-chip", has_text="kubernetes.md") - ).to_have_count(1) + page.locator(".msg.brain").last.locator(".source-chip") + ).to_have_count(0) def test_deflected_followup_receives_history( diff --git a/tests/e2e/test_llm_retry.py b/tests/e2e/test_llm_retry.py index 2413803..64d670d 100644 --- a/tests/e2e/test_llm_retry.py +++ b/tests/e2e/test_llm_retry.py @@ -393,18 +393,18 @@ def test_dead_then_recovered_grounded( _assert_no_error_frames(frames) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False - assert [(s["source"], s["path"]) for s in done["sources"]] == [ - (SEED_SOURCE, SEED_PATH) - ] + # Phase 119 (LOCKED A1): the grounded turn read NOTHING, so + # done.sources is empty (the suggested seed doc never chips — the + # retired phase-118 A4 union is gone); the durable record below + # still carries it (LOCKED A3, untouched). + assert done["sources"] == [], done["sources"] - # The grounded answer completed with the source chip — the agent - # round retried and the turn is intact. + # The grounded answer completed (no chips on a zero-read turn — + # phase 119 A1) — the agent round retried and the turn is intact. bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text(GROUNDED_Q) expect(bubble).to_contain_text(MOCK_ANSWER_MARKER) - chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(1) - expect(chips.first).to_contain_text(SEED_SP) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) _assert_no_error_banner(page) row = _last_query_log() @@ -446,12 +446,12 @@ def test_embedding_retry_completes( done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False - # The turn completed normally with the grounded answer + chip. + # The turn completed normally with the grounded answer (zero chips + # — the turn read nothing, so the chip row is empty under phase + # 119, LOCKED A1; the retired phase-118 A4 union is gone). bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text(MOCK_ANSWER_MARKER) - chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(1) - expect(chips.first).to_contain_text(SEED_SP) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) _assert_no_error_banner(page) diff --git a/tests/e2e/test_long_answers.py b/tests/e2e/test_long_answers.py index 4d4a28d..a34b67b 100644 --- a/tests/e2e/test_long_answers.py +++ b/tests/e2e/test_long_answers.py @@ -127,8 +127,8 @@ def test_normal_answer_unaffected( bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text("Deterministic mock answer for E2E", timeout=30_000) expect(bubble).not_to_contain_text("LONG-ANSWER-END") - # Grounded: the question's own document is cited as a chip. - expect( - page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - ).to_have_count(1, timeout=30_000) + # Grounded (the bubble + marker above) — and ZERO citation chips: + # the turn read nothing, so (phase 119, LOCKED A1) the chip row is + # empty (the retired phase-118 A4 suggested-chip is gone). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) expect(page.locator("#send-btn")).to_be_enabled() diff --git a/tests/e2e/test_ls_tree_drilldown.py b/tests/e2e/test_ls_tree_drilldown.py index a6e16e7..55f787f 100644 --- a/tests/e2e/test_ls_tree_drilldown.py +++ b/tests/e2e/test_ls_tree_drilldown.py @@ -678,8 +678,8 @@ def test_drill_down_sources_folders_files_and_read( expect(bubble).to_contain_text(READ_ANSWER_QUOTE) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False, done - # The read document is in the turn's sources (retrieval + agent-read, - # deduped — the grounded-turn record). + # The read document is in the turn's sources (phase 119, LOCKED A1: + # done.sources is the READ docs only — this turn's one read). assert any( s["path"] == TWO_A and s["source"] == ALPHA for s in done["sources"] ), done["sources"] diff --git a/tests/e2e/test_markdown_tables.py b/tests/e2e/test_markdown_tables.py index 389c531..50f83b5 100644 --- a/tests/e2e/test_markdown_tables.py +++ b/tests/e2e/test_markdown_tables.py @@ -185,9 +185,11 @@ def test_chat_table_renders( assert "|---|" not in bubble_text, "the |---| separator leaked into the bubble" assert "| Service | Port | Host |" not in bubble_text, "the raw header row leaked" - # Grounded retrieval: the table fixture is the top source chip. - chip = page.locator(".msg.brain .source-chip", has_text="homelab/tables.md") - expect(chip).to_have_count(1) + # Grounded (the table answer is the mock's non-deflected table + # branch) — and ZERO citation chips: the turn read nothing, so + # (phase 119, LOCKED A1) the chip row is empty (the retired + # phase-118 A4 suggested-chip is gone). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # --------------------------------------------------------------------------- diff --git a/tests/e2e/test_name_signal_read_chips.py b/tests/e2e/test_name_signal_read_chips.py new file mode 100644 index 0000000..9251505 --- /dev/null +++ b/tests/e2e/test_name_signal_read_chips.py @@ -0,0 +1,794 @@ +"""Phase 119 E2E (Playwright, mock-only): the name-signal seeds the +right document — and chips cite READ docs only. + +Run in isolation (DB must be up: ``podman compose up -d db``): + + uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov + +MOCK-ONLY suite: the story's gates are the deterministic contracts — +the mock's scripted-turn lenses (the tail echo, the single-read tool +flow, the new ``FOLDER_MAP_TRIGGER`` echo) make "what reached the +prompt / what got cited" assertable byte-exactly. + +KB fixture — ``tests/fixtures/namekb`` (tracked; registered as a +local-directory source through the authenticated API + the REAL +in-process ``POST /api/sync``, the ``test_summary_seed_context.py`` +pattern), seven markdown documents in one source (``namekb``) whose +bodies are token-controlled so the hybrid gate + the phase-119 +name-hit bonus pick the intended tiers deterministically +(``synced_kb`` pins it with the app's REAL retrieval — a +fixture-text regression that breaks the design fails at setup with a +clear message, not at the wire assertions): + +* every path carries a DIGITLESS question name token (the four + guides' stems hold ``install``, the two forgejo docs' stems + + folder hold ``forgejo``, ``zz-folder-map.md`` holds ``folder``) — + so the name-hit bonus applies to ALL seven docs (a uniform + ``+0.005``) and the tier order reduces to the pure retrieval + signal (the bonus head-start is exercised, the ordering is not + distorted by it); +* the four guides' bodies repeat the question phrase a strength + gradient (×4/×3/×2/×1) so they OUT-RANK the name-hit docs on + vector/FTS; the forgejo docs and the filler doc carry NEUTRAL + bodies (their name is their main signal — the phase-119 owner + scenario: "the file's name is the distinctive part"); +* engineered invariant (asserted in ``synced_kb`` for ALL FOUR + scripted questions, measured + probe-verified, stable across + re-imports): the name-hit doc ``forgejo/forgejo-home.md`` is the + LAST of the top-5 suggested (the name signal lifts it into the + tier, the weak body keeps it at the back — its ``Source:`` tail + line is the tail echo), and the related tier is the rank-6+ + remainder (``forgejo-nginx`` + ``zz-folder-map``; for the folder + question the two swap order — both stay rank 6+). + +Test → contract mapping (task 06 cases a–d; one Playwright file per +story, A16): +1. ``test_name_hit_doc_is_last_suggested`` — (a): the distinctive + question + ``show the end of your notes`` ⇒ the answer quotes the + last 160 chars of the seeded ```` block — the + name-hit doc's SUMMARY tail (its ``Source: namekb/forgejo/ + forgejo-home.md`` pointer line; no other doc's pointer line, no + doc's tail sentinel — the summary seed, not the full text). The + zero-read grounded turn chips NOTHING (LOCKED A1); the related + row renders rank 6+; the durable record carries suggested + + related (118-A3 untouched). +2. ``test_single_read_chips_only_the_read_doc`` — (b): the + ``use your tools`` flow (the mock's single read: ``ls`` → drill + ``ls(namekb)`` → read the FIRST file line — + ``namekb/zz-folder-map.md``, the root file) ⇒ the bubble carries + EXACTLY ONE ``.source-chip`` — the read doc (LOCKED A1); none of + the five suggested docs chips; the de-emphasized related row + renders (rank 6+ deduped against the cited read doc — the read + doc is a chip, never a "nearby doc"); the durable record carries + suggested + related + read (118-A3 untouched). +3. ``test_zero_read_grounded_turn_chips_nothing`` — (c): a PLAIN + distinctive question (no tool trigger) ⇒ the ``done`` frame's + ``sources == []`` and ZERO ``.source-chip`` elements in the + bubble — LOCKED A1's visible consequence; the answer still + renders grounded (deflected: false); the related row renders. +4. ``test_folder_lines_echoed_verbatim`` — (d): the distinctive + question + ``repeat your folder map`` (the new mock trigger) ⇒ + the answer ends with the suggested-folder lines VERBATIM — the + name-hit doc's folder line ``namekb/forgejo/: forgejo-nginx.md`` + (the line prefix + its known sibling entry) alongside the + guides' folder line. +""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +import time +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import httpx +import pytest +from playwright.sync_api import Locator, Page, expect +from sqlalchemy import select, text + +from app.config import Settings as _Settings +from app.db import SessionLocal +from app.models import Document, QueryLog +from app.rag.retriever import TRUNCATION_MARKER, retrieve, select_related, select_suggested +from e2e.auth_helpers import login +from e2e.conftest import ( + ADMIN_PASSWORD, + SESSION_SECRET, + USE_REAL_LLM, + _wait_http, +) +from e2e.mock_llm import TOKEN_RE, embed_text + +REPO = Path(__file__).resolve().parents[2] + +# Phase 79 (task 04, full inventory): the conftest session app owns its +# port in a combined run — this module app binds its own port instead +# (a same-port second uvicorn dies on bind and would drive the wrong +# server). Env-overridable. +APP_PORT = int(os.environ.get("E2E_APP_PORT_NAMESIG", "8145")) +APP_URL = f"http://127.0.0.1:{APP_PORT}" + +SOURCE = "namekb" # the local directory's basename = the source name +FIXTURES = REPO / "tests" / "fixtures" / "namekb" +MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E" + +# -------------------------------------------------------------------------- +# Fixture documents (tracked — tests/fixtures/namekb; deterministic, +# token-controlled — see the module docstring for the design) +# -------------------------------------------------------------------------- + +#: 22 neutral tokens (no question tokens) — the digest + filler +#: material of the neutral docs; the guides' strength phrase is +#: question-token-only. +PRELUDE = ( + "fixture preamble block one two three four five six seven eight nine ten " + "eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen" +) +assert len(TOKEN_RE.findall(PRELUDE.lower())) == 22 +PHRASE = "install forgejo with the deployment steps" + +#: (path, title, phrase repeats, filler line, tail sentinel) — the +#: guides carry the strength gradient (×4/×3/×2/×1); the forgejo docs +#: (name hits — distinctive digitless ``forgejo`` component) and the +#: root filler carry NEUTRAL bodies (0 repeats). +DOCS: list[tuple[str, str, int, str, str]] = [ + ("deploy/install-guide-a.md", "Install Forgejo Guide A", 4, + "alpha1 alpha2 alpha3 alpha4 alpha5 alpha6", "NAMEKB-TAIL-c5d6"), + ("deploy/install-guide-b.md", "Install Forgejo Guide B", 3, + "beta1 beta2 beta3 beta4 beta5 beta6", "NAMEKB-TAIL-d7e8"), + ("deploy/install-guide-c.md", "Install Forgejo Guide C", 2, + "gamma1 gamma2 gamma3 gamma4 gamma5 gamma6", "NAMEKB-TAIL-f9a0"), + ("deploy/install-guide-d.md", "Install Forgejo Guide D", 1, + "delta1 delta2 delta3 delta4 delta5 delta6", "NAMEKB-TAIL-b1c2"), + ("forgejo/forgejo-home.md", "Zeta Forge One", 0, + "kilo1 kilo2 kilo3 kilo4 kilo5 kilo6", "NAMEKB-TAIL-a1b2"), + ("forgejo/forgejo-nginx.md", "Zeta Forge Two", 0, + "lambda1 lambda2 lambda3 lambda4 lambda5 lambda6", "NAMEKB-TAIL-b3c4"), + ("zz-folder-map.md", "Zeta Misc Three", 0, + "micro1 micro2 micro3 micro4 micro5 micro6", "NAMEKB-TAIL-e3f4"), +] + +#: The engineered invariant (measured, probe-verified, stable across +#: re-imports — pinned for ALL FOUR scripted questions): the name-hit +#: doc is the LAST of the top-5 suggested; the guides lead in +#: retrieval-strength order (the ×2/×1 docs swap under the md5 +#: collision noise — pinned as measured, not as the gradient order). +SUGGESTED = [ + "deploy/install-guide-a.md", + "deploy/install-guide-b.md", + "deploy/install-guide-d.md", + "deploy/install-guide-c.md", + "forgejo/forgejo-home.md", +] +NAME_HIT = "forgejo/forgejo-home.md" # the (a) tail-echo target +RELATED = ["forgejo/forgejo-nginx.md", "zz-folder-map.md"] +#: The folder question's rank-6+ order (the filler's name hit — +#: ``folder`` — lifts it above the sibling; both stay rank 6+). +RELATED_FOLDER = ["zz-folder-map.md", "forgejo/forgejo-nginx.md"] +READ_TARGET = "zz-folder-map.md" # the single-read's first file line +SENTINELS = [sentinel for _p, _t, _i, _f, sentinel in DOCS] +ALL_SOURCE_LINES = [f"Source: {SOURCE}/{p}" for p, _t, _i, _f, _s in DOCS] + +#: The turn's questions (the mock's trigger phrases — see the module +#: docstring). The base question's name tokens (``install``, +#: ``forgejo``) ground every turn (best cosine ≈ 0.35–0.41 ≥ the E2E +#: 0.30 threshold) and fire the D1 name hits. +BASE_QUESTION = "How do I install forgejo?" +TAIL_QUESTION = BASE_QUESTION + " show the end of your notes" +TOOLS_QUESTION = BASE_QUESTION + " use your tools" +FOLDER_QUESTION = BASE_QUESTION + " repeat your folder map" +assert "show the end of your notes" in TAIL_QUESTION.lower() +assert "use your tools" in TOOLS_QUESTION.lower() +assert "repeat your folder map" in FOLDER_QUESTION.lower() + + +def _expected_summary(content: str, path: str) -> str: + """The mock lite model's byte-stable digest + the code pointer line + (mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch — + first 24 tokens of the document content — plus the summarizer's + deterministic ``Source:`` line; no model output is ever trusted).""" + digest = " ".join(TOKEN_RE.findall(content.lower())[:24]) + return f"This document covers {digest}.\nSource: {SOURCE}/{path}" + + +# -------------------------------------------------------------------------- +# Fixtures +# -------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def seed_dirs() -> Path: + """The story's local-directory source: the tracked fixture dir + (the app server runs on the same host, so the path is visible to + it). The directory's basename is the source name (``kind=local``, + phase 38).""" + assert FIXTURES.is_dir(), "tests/fixtures/namekb is missing" + for path, _t, _i, _f, _s in DOCS: + assert (FIXTURES / path).is_file(), f"fixture doc {path} is missing" + return FIXTURES + + +@pytest.fixture(scope="module") +def app_server(mock_llm: int, seed_dirs: Path) -> Iterator[str]: + """The real app under test — per-module app (the conftest pattern, + cf. ``test_summary_seed_context.py``): NO ``BOR_GIT_SOURCES`` (the + env fallback is git-only — the source here is a DB-registered local + directory), the mock LLM, the mock-calibrated threshold, and the + leak-guarded code defaults (the suggested/related tier settings + ride their code defaults — 5 / 2 — exactly like the production + ``.env``-free defaults). The session app is never started in this + isolated run, so no port clash.""" + env = dict(os.environ) + env.pop("DEBUGPY", None) + env["BOR_ENVIRONMENT"] = "e2e" + env["BOR_STATIC_DIR"] = str(REPO / "frontend") + env["BOR_LLM_BASE_URL"] = ( + "https://aipi.reeseapps.com/v1" + if USE_REAL_LLM + else f"http://127.0.0.1:{mock_llm}/v1" + ) + # Mock-calibrated gate (conftest pattern): the strength gradient + # keeps every scripted turn grounded (best cosine ≈ 0.35–0.41). + env["BOR_RELEVANCE_THRESHOLD"] = "0.30" + env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" + # Phase 67: instant retry waits + the code-default budget (the + # conftest leak-guard pattern). + env["BOR_LLM_RETRY_DELAY"] = "0" + env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default) + env.setdefault( + "BOR_DATABASE_URL", + "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", + ) + # Phase 16: admin auth must be set or create_app() refuses to boot. + env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD + env["BOR_SESSION_SECRET"] = SESSION_SECRET + # The repo's .env file carries the owner's BOR_GIT_SOURCES (the app + # reads it from cwd) — override it with an EMPTY value (the env var + # beats the .env file): the registry must hold EXACTLY the local + # directory this suite registers (a leftover env git list would + # pollute the KB the scripted turns run against). + env["BOR_GIT_SOURCES"] = "" + # Leak guards (conftest pattern): an operator's local (gitignored) + # .env cannot leak corpus-specific settings into the app under test. + env["BOR_DOCS_REPO"] = "" + env["BOR_SUGGESTIONS"] = json.dumps( + _Settings.model_fields["suggestions"].default + ) + env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default + env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default + proc = subprocess.Popen( + [sys.executable, "-m", "uvicorn", "app.main:app", + "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], + cwd=REPO, + env=env, + ) + try: + _wait_http(f"{APP_URL}/api/health") + yield APP_URL + finally: + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + + +@pytest.fixture(scope="module") +def app_url(app_server: str) -> str: + return app_server + + +def _truncate_all() -> None: + """Fresh registry + KB (the E2E isolation pattern): the E2E suites + share one Postgres, so a leftover git_sources row or document would + pollute the retrieval the scripted turns run against (the + name-signal margins are pinned against EXACTLY these seven + documents).""" + with SessionLocal() as db: + db.execute( + text( + "TRUNCATE chunks, documents, query_log, steering_notes, " + "kb_overview, git_sources, folder_summaries" + ) + ) + db.commit() + + +def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]: + """Poll the (cookie-authenticated) status endpoint until the run + reaches a terminal state (the test_summary_seed_context pattern, + over plain httpx — this fixture has no browser page yet).""" + deadline = time.monotonic() + timeout_s + body: dict[str, Any] = {} + while time.monotonic() < deadline: + r = client.get("/api/sync/status") + assert r.status_code == 200, r.text + body = r.json() + if body["state"] in ("success", "failed"): + return body + time.sleep(0.5) + raise AssertionError(f"sync did not reach a terminal state: {body}") + + +def _assert_tiers( + question: str, related_paths: list[str] +) -> None: + """Pin the name-signal design with the app's REAL hybrid retrieval + over the mock's embeddings (deterministic): the suggested tier is + exactly the four guides (retrieval-strength order) + the name-hit + doc LAST (LOCKED A3 — top-5, NO floor — the name-hit bonus, D2, + lifts it into the tier) and the related tier is the rank-6+ + remainder (``related_max_docs`` = 2). A fixture-text regression + that breaks the design fails here, at setup, with a clear + message.""" + with SessionLocal() as db: + chunks = retrieve(db, question, embed_text(question)) + suggested = [f"{d.source}/{d.path}" for d in select_suggested(chunks)] + related = [ + f"{d.source}/{d.path}" + for d in select_related( + chunks, + {d.id for d in select_suggested(chunks)}, + _Settings.model_fields["related_max_docs"].default, + ) + ] + expected = [f"{SOURCE}/{p}" for p in SUGGESTED] + assert suggested == expected, ( + f"suggested tier drifted for {question!r}: {suggested} " + f"(expected {expected})" + ) + assert related == [f"{SOURCE}/{p}" for p in related_paths], ( + f"related tier drifted for {question!r}: {related}" + ) + + +@pytest.fixture(scope="module") +def synced_kb(app_server: str, seed_dirs: Path) -> None: + """The story's precondition: the one-source KB synced under the + deterministic mock. Registers the fixture directory through the + authenticated API (the ``test_local_directory_sources.py`` + pattern), runs the REAL in-process sync (``POST /api/sync`` — walk + → chunk → embed → summaries → overview → folder summaries → + version bump), pins the stored content byte-identical to the + tracked fixture files, pins LOCKED A2 end-to-end (every doc stores + the mock's byte-stable digest + exactly one embedded + ``is_summary`` chunk), and pins the engineered invariant (the + name-hit doc LAST of the top-5) for ALL FOUR scripted questions.""" + _truncate_all() + with httpx.Client(base_url=app_server, timeout=30.0) as client: + r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) + assert r.status_code == 204, r.text + r = client.post( + "/api/git-sources", json={"kind": "local", "path": str(seed_dirs)} + ) + assert r.status_code == 201, r.text + r = client.post("/api/sync") + assert r.status_code == 202, r.text + body = _wait_sync_done_http(client) + assert body["state"] == "success", body + detail = body["detail"] + assert detail["added"] == len(DOCS), detail + assert detail["pruned"] == 0, detail + # The import stored the tracked fixture strings BYTE-IDENTICALLY + # and, for EVERY doc, the mock's byte-stable digest: the + # deterministic assertion surface of the whole suite. + with SessionLocal() as db: + for path, _title, _i, _f, _s in DOCS: + on_disk = (FIXTURES / path).read_text(encoding="utf-8") + stored = db.scalar( + select(Document).where( + Document.source == SOURCE, Document.path == path + ) + ) + assert stored is not None, f"fixture doc {path} was not imported" + assert stored.content == on_disk, f"stored content drifted for {path}" + assert stored.summary == _expected_summary(on_disk, path), ( + f"summary is not the mock digest for {path}: {stored.summary!r}" + ) + schunks = [c for c in stored.chunks if c.is_summary] + assert len(schunks) == 1 and schunks[0].position == -1, ( + f"expected exactly one is_summary chunk for {path}" + ) + assert schunks[0].embedding is not None, ( + f"the is_summary chunk of {path} is not embedded" + ) + _assert_tiers(TAIL_QUESTION, RELATED) + _assert_tiers(TOOLS_QUESTION, RELATED) + _assert_tiers(FOLDER_QUESTION, RELATED_FOLDER) + _assert_tiers(BASE_QUESTION, RELATED) + + +@pytest.fixture(autouse=True) +def _clean(db_ready: None) -> Iterator[None]: + """Per-test query_log isolation (the KB itself is module-scoped — + the scripted turns never change it, so the registry and the KB + persist across the tests of this module).""" + with SessionLocal() as db: + db.execute(text("TRUNCATE query_log")) + db.commit() + yield + with SessionLocal() as db: + db.execute(text("TRUNCATE query_log")) + db.commit() + + +# -------------------------------------------------------------------------- +# Page helpers (the test_summary_seed_context house patterns) +# -------------------------------------------------------------------------- + +#: Captures the raw SSE ``data:`` payloads of the /api/chat stream +#: (a response clone read in the background) — wire-level assertions +#: for the ``tool`` / ``done`` frames, independent of the UI rendering. +SSE_HOOK = """ +() => { + if (window.__sseInstalled) return; + window.__sseInstalled = true; + window.__sseFrames = []; + const origFetch = window.fetch; + window.fetch = async function (...args) { + const res = await origFetch.apply(this, args); + try { + const url = typeof args[0] === 'string' ? args[0] : args[0].url; + if (url.includes('/api/chat')) { + res.clone().text().then((bodyText) => { + for (const block of bodyText.split('\\n\\n')) { + const line = block.trim(); + if (line.startsWith('data: ')) { + window.__sseFrames.push(line.slice(6)); + } + } + }); + } + } catch (e) { /* non-clonable responses: ignored */ } + return res; + }; +} +""" + + +def _install_page_hooks(page: Page) -> None: + page.evaluate(SSE_HOOK) + + +def _frames(page: Page) -> list[dict]: + """The SSE frames captured since the last submit (``_submit`` + clears the buffer), once the hook's background read settles.""" + deadline = time.monotonic() + 30.0 + while True: + raw = page.evaluate("() => window.__sseFrames || []") + parsed = [json.loads(line) for line in raw if line] + if any(f.get("type") == "done" for f in parsed): + return parsed + if time.monotonic() > deadline: + raise AssertionError( + f"SSE hook captured no `done` frame (frames so far: " + f"{len(parsed)}) — hook install failed?" + ) + time.sleep(0.05) + + +def _tool_frames(frames: list[dict]) -> list[dict]: + return [f for f in frames if f.get("type") == "tool"] + + +def _submit(page: Page, question: str) -> None: + page.evaluate("window.__sseFrames = []") + page.fill("#message-input", question) + page.click("#send-btn") + # The user bubble lands synchronously with the submit handler. + expect(page.locator(".msg.user .bubble").last).to_contain_text(question) + + +def _wait_settled(page: Page) -> None: + """The turn is complete: answer text in the bubble, button recovered + (the phase-48 settle wait).""" + expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000) + expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000) + expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) + + +def _last_brain(page: Page) -> Locator: + return page.locator(".msg.brain").last + + +def _last_query_log() -> QueryLog: + with SessionLocal() as db: + rows = db.scalars(select(QueryLog)).all() + assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}" + return rows[0] + + +def _assert_chip_row(page: Page, read_paths: list[str]) -> None: + """The citation surface of a grounded turn (phase 119, LOCKED A1 — + the retired phase-118 A4 union is gone): the chip row is the + AGENT-READ docs only (a zero-read turn chips NOTHING: the + suggested docs are seed context, not citations).""" + chips = page.locator(".msg.brain .source-chip") + expect(chips).to_have_count(len(read_paths)) + for path in read_paths: + expect( + chips.filter(has_text=path), message=f"chip for {path}" + ).to_have_count(1) + + +def _assert_related_row(page: Page, related_paths: list[str]) -> None: + """The de-emphasized ``related-docs`` row (phase-113 UI, untouched + by phase 119 — never a citation chip): the rank-6+ remainder + deduped against the cited read docs (a read related doc is a chip, + never a "nearby doc").""" + row = page.locator(".msg.brain .related-docs") + expect(row).to_have_count(1) + expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case") + links = page.locator(".msg.brain .related-docs .related-doc") + expect(links).to_have_count(len(related_paths)) + for i, path in enumerate(related_paths): + expect(links.nth(i)).to_contain_text(f"{SOURCE}/{path}") + # The related links are never citation chips. + expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0) + + +# -------------------------------------------------------------------------- +# 1. (a) the name-signal doc is the LAST of the top-5 suggested — the +# tail echo proves it reached the prompt as the last seed block +# -------------------------------------------------------------------------- + + +def test_name_hit_doc_is_last_suggested( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """The tail-echo question quotes the last 160 chars of the seeded + ```` block: they end in the LAST suggested doc's SUMMARY + — the name-hit doc's byte-stable digest tail + pointer line (the + digitless-component rule + the D2 bonus put ``forgejo-home`` in + the tier, its neutral body keeps it last). No other doc's pointer + line and no doc's tail sentinel are in the echoed context — the + summary seed, not the full text. The zero-read grounded turn + chips nothing (LOCKED A1); the related row renders rank 6+; the + durable record carries suggested + related (118-A3 untouched).""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, TAIL_QUESTION) + _wait_settled(page) + + # No tools on this turn — the answer is the mock's direct tail echo + # of the seeded context (the summary-seed lens). + frames = _frames(page) + assert _tool_frames(frames) == [], _tool_frames(frames) + done = next(f for f in frames if f.get("type") == "done") + assert done["deflected"] is False, done + # Phase 119 (LOCKED A1): the grounded done frame cites the READ + # docs only — this turn read NOTHING (the summary-only fast path), + # so the citation surface is empty; the suggested set (name-hit + # doc included) is seed context, not citations. + assert done["sources"] == [], done["sources"] + + bubble = _last_brain(page).locator(".bubble") + expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) + + # The echoed tail ends in the name-hit doc's summary: its + # deterministic ``Source:`` pointer line (the digest is pinned in + # ``synced_kb`` — a content preview would carry the neutral-body + # text instead, and the pointer line only ever exists on a stored + # summary). The bubble renders the answer as markdown, which + # collapses the summary's newline — pin the pointer line alone + # (it sits inside the echoed 160 chars, byte-exact). + expect(bubble).to_contain_text(f"Source: {SOURCE}/{NAME_HIT}") + # No OTHER doc's pointer line is in the echoed tail — the + # name-hit doc was the LAST suggested block (the (a) invariant's + # visible proof). + for path, _t, _i, _f, _s in DOCS: + if path != NAME_HIT: + expect(bubble).not_to_contain_text(f"Source: {SOURCE}/{path}") + # And no document's tail sentinel: the full content of no + # suggested doc reached the model (summary seed only). + for sentinel in SENTINELS: + expect(bubble).not_to_contain_text(sentinel) + expect(bubble).not_to_contain_text(TRUNCATION_MARKER) + + # The zero-read grounded turn chips NOTHING (LOCKED A1); the + # related row renders rank 6+ (the de-emphasized row, untouched). + _assert_chip_row(page, []) + _assert_related_row(page, RELATED) + + # Durable record: grounded; suggested + related (LOCKED A3 — the + # log records retrieval, not citations; phase 119 A1 retires the + # chip surface only, not the record). + row = _last_query_log() + assert row.question == TAIL_QUESTION + assert row.deflected is False + assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED) + + +# -------------------------------------------------------------------------- +# 2. (b) chips = READ docs only — the single-read flow chips exactly +# the one read doc; the related row renders (deduped) +# -------------------------------------------------------------------------- + + +def test_single_read_chips_only_the_read_doc( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """The ``use your tools`` flow drives the mock's single read: + ``ls`` (the top level) → drill ``ls(namekb)`` → ``read`` the FIRST + file line (the root file ``zz-folder-map.md`` — subfolders list + first, files after). The bubble carries EXACTLY ONE + ``.source-chip`` — the read doc (LOCKED A1); none of the five + suggested docs chips (the name-hit doc is seed context, not a + citation); the de-emphasized related row renders the rank-6+ + remainder deduped against the cited read doc (the read doc is a + chip, never a "nearby doc").""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, TOOLS_QUESTION) + _wait_settled(page) + + # Wire level: exactly three `tool` frames — ``ls`` (the top + # level), the drill ``ls`` scoped to the first (only) source + # (phase 94), then ``read`` the first file line's combined + # source/path — and all three ahead of the first `delta` frame. + frames = _frames(page) + assert _tool_frames(frames) == [ + {"type": "tool", "name": "ls", "argument": None}, + {"type": "tool", "name": "ls", "argument": SOURCE}, + {"type": "tool", "name": "read", "argument": f"{SOURCE}/{READ_TARGET}"}, + ], _tool_frames(frames) + first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta") + assert all( + i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool" + ) + done = next(f for f in frames if f.get("type") == "done") + assert done["deflected"] is False, done + # Phase 119 (LOCKED A1): done.sources = the READ docs only — + # exactly the one read; the five suggested docs (name-hit doc + # included) never chip. + assert [(s["source"], s["path"]) for s in done["sources"]] == [ + (SOURCE, READ_TARGET) + ], done["sources"] + # done.related = the rank-6+ remainder deduped against the cited + # read doc — the read doc is rank 7 (IN the remainder) and drops + # out: only the sibling name-hit doc remains. + assert [(s["source"], s["path"]) for s in done["related"]] == [ + (SOURCE, "forgejo/forgejo-nginx.md") + ], done["related"] + + # The final answer quotes the read document (the mock's + # deterministic quote: "Read . "). + bubble = _last_brain(page).locator(".bubble") + expect( + bubble, + ).to_contain_text(f"Read {SOURCE}/{READ_TARGET}.", timeout=30_000) + expect(bubble).not_to_contain_text(TRUNCATION_MARKER) + + # The UI chip row: EXACTLY the one read doc (LOCKED A1) — and + # explicitly NONE of the suggested docs (the name-hit doc first). + _assert_chip_row(page, [READ_TARGET]) + expect( + page.locator(".msg.brain .source-chip", has_text="install-guide") + ).to_have_count(0) + expect( + page.locator(".msg.brain .source-chip", has_text=NAME_HIT) + ).to_have_count(0) + # The related row renders rank 6+ deduped against the cited read + # doc (phase-113 behavior untouched). + _assert_related_row(page, ["forgejo/forgejo-nginx.md"]) + + # Durable record: grounded; suggested + related + read (deduped — + # the read doc sits in the related slot, LOCKED A3 — phase 119 A1 + # retires the chip surface only, not the record). + row = _last_query_log() + assert row.question == TOOLS_QUESTION + assert row.deflected is False + assert row.sources == ", ".join( + f"{SOURCE}/{p}" for p in SUGGESTED + RELATED + ) + + +# -------------------------------------------------------------------------- +# 3. (c) a zero-read grounded turn chips nothing (LOCKED A1's visible +# consequence) +# -------------------------------------------------------------------------- + + +def test_zero_read_grounded_turn_chips_nothing( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """A PLAIN distinctive question (no tool trigger) grounds on the + name tokens: the ``done`` frame's ``sources == []`` and ZERO + ``.source-chip`` elements in the bubble — LOCKED A1's visible + consequence (the suggested docs, name-hit doc included, are seed + context, never citations); the answer still renders grounded + (deflected: false); the related row renders rank 6+.""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, BASE_QUESTION) + _wait_settled(page) + + frames = _frames(page) + assert _tool_frames(frames) == [], _tool_frames(frames) + done = next(f for f in frames if f.get("type") == "done") + assert done["deflected"] is False, done + # LOCKED A1's wire-level consequence: a grounded turn that read + # nothing cites nothing. + assert done["sources"] == [], done["sources"] + + bubble = _last_brain(page).locator(".bubble") + expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) + # Zero chips — not just "no suggested chips": the citation + # surface is empty. + _assert_chip_row(page, []) + _assert_related_row(page, RELATED) + + # Durable record: grounded; suggested + related (LOCKED A3). + row = _last_query_log() + assert row.question == BASE_QUESTION + assert row.deflected is False + assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED) + + +# -------------------------------------------------------------------------- +# 4. (d) the suggested-folder lines ride the prompt — echoed verbatim +# by the new mock trigger +# -------------------------------------------------------------------------- + + +def test_folder_lines_echoed_verbatim( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """The distinctive question + ``repeat your folder map`` (the new + mock trigger — the prompt-injection-echo convention of + ```` / ````) ⇒ the answer ends with the + suggested-folder lines VERBATIM (joined with ``; ``): the guides' + folder line (its owner excluded from the entries) and the + name-hit doc's folder line — ``namekb/forgejo/: forgejo-nginx.md`` + (the line prefix + its known sibling entry; the owner + ``forgejo-home`` is excluded — its identity is already in its + ```` block).""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, FOLDER_QUESTION) + _wait_settled(page) + + frames = _frames(page) + assert _tool_frames(frames) == [], _tool_frames(frames) + done = next(f for f in frames if f.get("type") == "done") + assert done["deflected"] is False, done + assert done["sources"] == [], done["sources"] + + bubble = _last_brain(page).locator(".bubble") + expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) + + # The folder lines VERBATIM — both lines, suggested-folder order + # (the guides' folder first — its first suggested doc leads; the + # name-hit doc's folder second), entries owner-excluded. + expect(bubble).to_contain_text( + "(folders: " + "namekb/deploy/: install-guide-b.md, install-guide-c.md, install-guide-d.md; " + "namekb/forgejo/: forgejo-nginx.md)", + ) + # The task's pin: the name-hit doc's folder line prefix + one + # known sibling entry. + expect(bubble).to_contain_text(f"{SOURCE}/forgejo/: forgejo-nginx.md") + + # Zero-read grounded turn — chips nothing (LOCKED A1). + _assert_chip_row(page, []) + + # Durable record: grounded; suggested + related (LOCKED A3 — the + # folder question's rank-6+ order). + row = _last_query_log() + assert row.question == FOLDER_QUESTION + assert row.deflected is False + assert row.sources == ", ".join( + f"{SOURCE}/{p}" for p in SUGGESTED + RELATED_FOLDER + ) diff --git a/tests/e2e/test_no_reply_autoscroll.py b/tests/e2e/test_no_reply_autoscroll.py index bbe5da8..d1f3308 100644 --- a/tests/e2e/test_no_reply_autoscroll.py +++ b/tests/e2e/test_no_reply_autoscroll.py @@ -472,13 +472,13 @@ def test_answer_content_intact(page: Page, app_url: str, seeded_kb: None) -> Non page.set_default_timeout(30_000) login(page, app_url, next="/") - # The long answer streams to completion with its sources ... + # The long answer streams to completion (phase 119, LOCKED A1: a + # zero-read turn chips nothing — the suggested kubernetes doc never + # chips; the retired phase-118 A4 union is gone) ... submit(page, LONG_QUESTION) wait_settled(page) expect(page.locator(".msg.brain .bubble").last).to_contain_text(LONG_ANSWER_END) - expect( - page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - ).to_have_count(1) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # ... and a thinking turn completes with its block auto-collapsed # (phase 17: open while streaming, closed from the first delta on). @@ -489,25 +489,25 @@ def test_answer_content_intact(page: Page, app_url: str, seeded_kb: None) -> Non expect(details.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT) expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER) - # Persistence: four messages, the thinking text + sources stored raw. + # Persistence: four messages, the thinking text stored raw — and + # the thinking turn's sources EMPTY (phase 119, LOCKED A1: it read + # nothing, so the done frame — and the record — carry no sources; + # the retired phase-118 A4 union is gone). raw = page.evaluate(f"() => localStorage.getItem('{STORAGE_KEY}')") stored = json.loads(raw) assert [m["who"] for m in stored["messages"]] == ["user", "brain", "user", "brain"] assert LONG_ANSWER_END in stored["messages"][1]["text"] assert THINKING_FRAGMENT in stored["messages"][3]["thinking"] - assert any( - s["path"] == "homelab/kubernetes.md" for s in stored["messages"][3]["sources"] - ) + assert stored["messages"][3]["sources"] == [] - # Restore: the long answer (with its chip) and the COLLAPSED thinking - # block come back intact. + # Restore: the long answer and the COLLAPSED thinking block come + # back intact — with ZERO citation chips (both turns read nothing; + # phase 119, LOCKED A1). page.reload() expect(page.locator(".msg.user .bubble")).to_have_count(2) expect(page.locator(".msg.brain .bubble")).to_have_count(2) expect(page.locator(".msg.brain .bubble").first).to_contain_text(LONG_ANSWER_END) - expect( - page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - ).to_have_count(2) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) restored = page.locator(".msg.brain").last.locator("details.thinking") expect(restored).not_to_have_attribute("open") expect(restored.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT) diff --git a/tests/e2e/test_quadlet_jinja_import.py b/tests/e2e/test_quadlet_jinja_import.py index 29d2270..0d1c4e7 100644 --- a/tests/e2e/test_quadlet_jinja_import.py +++ b/tests/e2e/test_quadlet_jinja_import.py @@ -24,8 +24,10 @@ Test → story mapping (Playwright Mapping Rule): 4. ``test_jinja_retrievable_not_deflected`` — a question carrying the ``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine below threshold **and** zero FTS hits) → honest-positive: the answer - bubble is not ``.is-deflected`` and a source chip names - ``templates/deploy.j2``. + bubble is not ``.is-deflected`` — and (phase 119, LOCKED A1) the + zero-read turn chips NOTHING (the ``templates/deploy.j2`` retrieval + doc is suggested context, not a chip; the retired phase-118 A4 + union is gone). Phase 97 adaptation: the Sources table is the DRILL-DOWN TREE — the rows live at their folder levels (``docs`` → ``homelab`` → ``quadlet`` @@ -218,10 +220,16 @@ def test_jinja_retrievable_not_deflected( page.fill("#message-input", JINJA_QUESTION) page.click("#send-btn") - # The done event appends source chips — waiting on the .j2 chip means - # the turn is finished and the retrieval doc reached the UI. - chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2") - expect(chip).to_have_count(1, timeout=30_000) + # Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation + # chips — the .j2 retrieval doc is suggested context, not a chip + # (the retired phase-118 A4 union is gone). The turn is finished + # (and the grounded, non-deflected state is reached) when the + # answer settles with the send button recovered. + expect(page.locator(".msg.brain .bubble").last).to_contain_text( + "Deterministic mock answer for E2E", timeout=30_000 + ) + expect(page.locator("#send-label")).to_have_text("Send") + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # A8: LOW requires best cosine < threshold AND zero FTS hits — the # question's sentinel tokens FTS-match the .j2 chunk, so the gate is diff --git a/tests/e2e/test_read_truncation_cap.py b/tests/e2e/test_read_truncation_cap.py index 7d68c82..833f9e5 100644 --- a/tests/e2e/test_read_truncation_cap.py +++ b/tests/e2e/test_read_truncation_cap.py @@ -658,10 +658,11 @@ def test_truncated_read_frame_order_live_marker_and_llm_notice( assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False, done - # The read document is the turn's cited source (retrieval + - # agent-read, deduped — the grounded-turn record; phase 118: the - # target may be a suggested (summary-seeded) document — its seed - # status never changes the cap's wire contract). + # The read document is in the turn's cited sources (phase 119, + # LOCKED A1: done.sources is the READ docs only — this turn's one + # read; phase 118: the target may be a suggested (summary-seeded) + # document — its seed status never changes the cap's wire + # contract). assert any( s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"] ), done["sources"] @@ -797,6 +798,8 @@ def test_short_read_control_no_frame_no_marker( assert _result_frames(frames) == [], _result_frames(frames) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False, done + # The read document is the turn's cited source (phase 119 A1 — the + # READ docs only; the target may be a suggested/summary-seeded doc). assert any( s["path"] == SHORT_REL and s["source"] == SOURCE for s in done["sources"] ), done["sources"] diff --git a/tests/e2e/test_retrieval_quality.py b/tests/e2e/test_retrieval_quality.py index 7ef9c08..f4c152e 100644 --- a/tests/e2e/test_retrieval_quality.py +++ b/tests/e2e/test_retrieval_quality.py @@ -10,11 +10,15 @@ deterministic mock embeddings (same pattern as the earlier story suites). The four tests map the story's acceptance criteria: 1. multi-format fixture import — hidden doc excluded, ``/api/docs`` counts -2. "How did I install gitlab?" — grounded (not deflected), gitlab chip, - ``query_log`` row with the gitlab doc in ``sources`` +2. "How did I install gitlab?" — grounded (not deflected), ZERO citation + chips (phase 119, LOCKED A1 — the turn reads nothing: the suggested + gitlab doc is context, not a chip), ``query_log`` row with the gitlab + doc in ``sources`` (LOCKED A3, untouched) 3. keyword-only question ("kafkabridge") beats the vector ranking — the corroborated-lexical gate (A8 revised 2026-09-14) grounds it end to - end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor + end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor; + the FTS-matched doc tops the durable record (the chip pin is retired + with the phase-118 A4 union — a zero-read turn chips nothing) 4. "sourdough" — deflected bubble + ≥2 "Maybe try" chips """ from __future__ import annotations @@ -156,9 +160,15 @@ def test_gitlab_question_is_grounded_with_gitlab_chip( # Grounded: no deflected bubble at all. expect(page.locator(".msg.brain.is-deflected")).to_have_count(0) - # The gitlab document is cited (a chip carrying its path). - chip = page.locator(".msg.brain .source-chip", has_text="container_gitlab/gitlab.md") - expect(chip).to_have_count(1, timeout=30_000) + # Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation + # chips — the suggested gitlab doc is seed context, not a citation + # (the retired phase-118 A4 union is gone); the ranking assertion + # lives in the durable record below (LOCKED A3, untouched). The + # button label is the settle sync: the client re-labels Send on the + # done frame, and the server writes the query_log row just before + # yielding it. + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) + expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) # Durable record: not deflected, and the gitlab doc is in sources. with SessionLocal() as db: @@ -186,10 +196,15 @@ def test_keyword_only_question_beats_vector_ranking( expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) expect(page.locator(".msg.brain.is-deflected")).to_have_count(0) - # The FTS-matched doc is the TOP source chip (it beats the vector rank). - first_chip = page.locator(".msg.brain .source-chip").first - first_chip.wait_for(state="visible", timeout=30_000) - expect(first_chip).to_contain_text("static-dns.json") + # Phase 119 (LOCKED A1): zero-read turn → ZERO chips (the + # FTS-matched doc beats the vector rank in the DURABLE record below + # — LOCKED A3, untouched; the retired phase-118 A4 chip pin is + # gone). Wait for the settle via the Send label instead of a chip + # (the in-flight button is the enabled Stop control — only the + # label proves the done frame landed and the query_log row was + # written). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) + expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) with SessionLocal() as db: row = db.scalars(select(QueryLog)).one() diff --git a/tests/e2e/test_search_tool.py b/tests/e2e/test_search_tool.py index b96a71a..a6af422 100644 --- a/tests/e2e/test_search_tool.py +++ b/tests/e2e/test_search_tool.py @@ -50,9 +50,11 @@ Test → phase mapping: (``Found …`` — the match reached the model), and the turn settles to idle with no error banner. 2. ``test_search_adds_no_source_by_itself`` — context accounting - (locked A5): the search-only flow (no read) leaves - ``done.sources`` / the source chips / ``query_log.sources`` at the - retrieval baseline — the search adds no source by itself. + (locked A5, phase-119 A1 shape): the search-only flow (no read) + leaves ``done.sources`` / the source chips EMPTY (chips cite read + docs only — the search adds no source by itself, and the retrieval + doc was never read) while ``query_log.sources`` keeps the retrieval + baseline (LOCKED A3, untouched). 3. ``test_search_tool_line_re_renders_after_reload`` — the persisted record (phase 14 convention: the generic ``{name, argument}`` toolAcc) re-renders the search line through the same helper. @@ -423,25 +425,24 @@ def test_search_adds_no_source_by_itself( _wait_settled(page) # The search really ran (its wire frame is present) — yet the - # search-only flow (no read) leaves done.sources at the RETRIEVAL - # baseline: the one fixture doc, nothing added by the search. + # search-only flow (no read) adds NO cited source: phase 119, + # LOCKED A1 — done.sources is the READ docs only, and nothing was + # read (the retrieval doc was never read, so it never chips — the + # retired phase-118 A4 suggested+read union is gone). frames = _frames(page) assert _tool_frames(frames) == [ {"type": "tool", "name": "grep", "argument": SEARCH_PATTERN} ] done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False - assert [(s["source"], s["path"]) for s in done["sources"]] == [ - (SEED_SOURCE, SEED_PATH) - ] + assert done["sources"] == [], done["sources"] - # UI: exactly one source chip — the retrieval doc (the search - # renders no chip of its own). - chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(1) - expect(chips.nth(0)).to_contain_text(SEED_SP) + # UI: ZERO source chips — the search renders no chip of its own, and + # the never-read retrieval doc chips nothing (phase 119 A1). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) - # Durable record: the sources row is unchanged by the search alone. + # Durable record: the sources row keeps the retrieval baseline (the + # search adds nothing to it either — LOCKED A3, untouched). row = _last_query_log() assert row.deflected is False assert row.sources == SEED_SP diff --git a/tests/e2e/test_share_chat.py b/tests/e2e/test_share_chat.py index 7cbc944..0114aa1 100644 --- a/tests/e2e/test_share_chat.py +++ b/tests/e2e/test_share_chat.py @@ -25,12 +25,14 @@ click is the idempotent share of the linked row): cookies) opening ``/shared/`` sees the full conversation read-only through the same record shape: title = the auto-title, user + brain bubbles (the same deterministic answer text the admin - session saw), the thinking block RESTORED COLLAPSED, the source - chips as PLAIN TEXT (zero ``a.source-chip`` — guests cannot open - documents, the documents API is admin-only), and ZERO interactive - controls anywhere (no composer, no Save/Share pills, no Tune/Retry, - no button chips); the nav's admin-only links stay hidden for a - guest; + session saw), the thinking block RESTORED COLLAPSED, ZERO source + chips of any kind — the shared turn read nothing, so (phase 119, + LOCKED A1) its ``done`` sources are empty and nothing re-renders as a + chip; where chips exist they are always PLAIN TEXT (zero + ``a.source-chip`` — guests cannot open documents, the documents API + is admin-only) — and ZERO interactive controls anywhere (no + composer, no Save/Share pills, no Tune/Retry, no button chips); the + nav's admin-only links stay hidden for a guest; * **Share from History + unshare** — the History row's Share column: "Create link" → Copy + Unshare; Unshare is the inline two-step (no native dialog); after Yes the cell returns to "Create link", @@ -345,17 +347,13 @@ def test_anonymous_shared_view( expect(think).to_have_count(1) expect(think.first).not_to_have_attribute("open") - # Source chips are PLAIN TEXT: the on-topic turn carries its - # source chips (the suggested docs — the hybrid retrieval's - # summary-seed tier, phase 118), but every one as a : - # zero anywhere (a guest cannot open documents; - # the documents API is admin-only). - assert ( - anon.locator(".msg.brain .source-chip").count() >= 1 - ), "the grounded turn must carry its source chips" - expect( - anon.locator(".msg.brain .source-chip", has_text="kubernetes.md") - ).to_have_count(1) + # Source chips: ZERO — the shared turn read nothing, so + # (phase 119, LOCKED A1) its done sources are empty and the + # shared page re-renders nothing (the retired phase-118 A4 + # suggested+read union is gone); and where chips DO exist they + # are always plain text: zero anywhere (a + # guest cannot open documents; the documents API is admin-only). + expect(anon.locator(".msg.brain .source-chip")).to_have_count(0) expect(anon.locator("a.source-chip")).to_have_count(0) # ZERO interactive controls anywhere in the conversation: no diff --git a/tests/e2e/test_source_chip_quality.py b/tests/e2e/test_source_chip_quality.py index b404821..3953953 100644 --- a/tests/e2e/test_source_chip_quality.py +++ b/tests/e2e/test_source_chip_quality.py @@ -1,22 +1,27 @@ -"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface -contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract -replaced the phase-112/113 usefulness bar: the chip row is the -SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on -these turns), the related row is rank 6+ (capped at -``related_max_docs`` = 2), and a deflected turn still cites nothing -(done.sources = [] — its weak hits are suggested for the durable record -but are never citation chips). +"""Phase 113 E2E (Playwright) — phase 118 re-targeted, phase 119 +re-targeted again: the citation-surface contract (phase 119, LOCKED +A1) as VISIBLE chip counts — chips are the AGENT-READ docs only (the +retired phase-118 A4 suggested+read union is gone): neither turn on +this suite reads anything, so a grounded turn shows ZERO citation +chips (the suggested tier is seed context, not citations — the +phase-112/113 usefulness bar was retired with the full-text seeds, and +the suggested chips were retired with the phase-118 A4 union), the +related row is rank 6+ (capped at ``related_max_docs`` = 2, untouched), +and a deflected turn still cites nothing (done.sources = [] — its weak +hits are suggested for the durable record but are never citation +chips). Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov -Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row -is the suggested tier — the 5 docs the model was seeded with (its -"start here" set — the old "one chip per bar-clearing doc" is retired -with the full-text seeds; the LLM decides what the summaries earn); -a deflected turn shows zero citation chips (its weak hits, if any, live -in the de-emphasized related row — ``.related-doc`` links, NEVER +Acceptance (TODO L144–146, phase-119 shape): a grounded turn's chip row +is the READ set — empty on these no-read turns (the visible consequence +of the owner decision 2026-09-16: the suggested docs the model was +seeded with are context, not citations — the LLM decides what the +summaries earn, and a zero-read turn chips nothing); a deflected turn +shows zero citation chips (its weak hits, if any, live in the +de-emphasized related row — ``.related-doc`` links, NEVER ``.source-chip``; L2c: never render weak hits as answer citations). The fixture KB's tier shapes are deterministic under the E2E mock's @@ -25,14 +30,20 @@ here against the wire): * **grounded question** — "What SSH aliases do I have?": best cosine 0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is - ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md - > backups.md (fused rank — the lexical-only cosine-0.0 docs rank when - they rank: no floor filters them, LOCKED A3) and the done frame - carries EXACTLY those five cited refs; the rank-6+ remainder - (compose.container, static-dns.json) rides the related row, capped at - two. (The four OBSERVED live shapes are unit-pinned at plan level in - ``tests/unit/test_source_chip_quality.py`` — the fixture KB - reproduces the tiered shape live, so no docstring caveat is needed.) + ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > backups.md + > kubernetes.md (fused rank — the lexical-only cosine-0.0 docs rank + when they rank: no floor filters them, LOCKED A3; phase 119, D1 — + ``aliases`` is a stem sub-component of ``ssh_aliases.txt``, so the + doc name-hits, its name-hit row LEADS the lexical list, and the FTS + rank shift re-orders the fused scores — the pre-phase + kubernetes-before-backups order is retired) and the done frame + carries NO cited refs (phase 119 A1 — nothing was read: the + suggested tier seeds the prompt but never chips); the rank-6+ + remainder (compose.container, static-dns.json) rides the related + row, capped at two. (The four OBSERVED live shapes are unit-pinned at + plan level in ``tests/unit/test_source_chip_quality.py`` — the + fixture KB reproduces the tiered shape live, so no docstring caveat + is needed.) * **deflected question** — "How do I bake sourdough bread?": best cosine 0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs (done.sources = []); the weak hits ARE suggested (no floor) for the @@ -106,14 +117,15 @@ def _ask(page: Page, message: str) -> None: def test_single_source_question_shows_exactly_one_citation_chip( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: - """The acceptance pin (phase 118, A4): a grounded question → the done - bubble carries EXACTLY the suggested tier as ``.source-chip``s (the - five docs the model was seeded with — top-5, NO floor: the old - usefulness-bar "one chip" is retired with the full-text seeds) and - the rank-6+ remainder renders only in the de-emphasized - ``.related-docs`` row (``.related-doc`` links — never - ``.source-chip``), labeled "Nearby docs, in case:". The durable - record keeps the FULL retrieval (LOCKED A3).""" + """The acceptance pin (phase 119, A1): a grounded, NO-READ question → + the done bubble carries ZERO ``.source-chip``s (the chip row is the + agent-read docs only — nothing was read here: the five suggested + docs the model was seeded with are context, not citations — the + retired phase-118 A4 union is gone) and the rank-6+ remainder + renders only in the de-emphasized ``.related-docs`` row + (``.related-doc`` links — never ``.source-chip``), labeled "Nearby + docs, in case:" (phase-113 behavior, untouched). The durable record + keeps the FULL retrieval (LOCKED A3, untouched).""" _reset_db(mock_llm) page.set_default_timeout(30_000) login(page, app_url, next="/") # phase 79: chat is require_user-gated @@ -127,16 +139,14 @@ def test_single_source_question_shows_exactly_one_citation_chip( # Grounded: no deflected bubble at all. expect(page.locator(".msg.brain.is-deflected")).to_have_count(0) - # The citation surface IS the suggested tier (LOCKED A4 — top-5, - # NO floor): five chips, in fused rank order — the model was - # seeded with exactly these five docs' summaries. - chip = page.locator(".msg.brain .source-chip") - expect(chip).to_have_count(5, timeout=30_000) - expect(chip.nth(0)).to_contain_text("ssh_aliases.txt") - expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml") - expect(chip.nth(2)).to_contain_text("uptime_probe.py") - expect(chip.nth(3)).to_contain_text("kubernetes.md") - expect(chip.nth(4)).to_contain_text("backups.md") + # Phase 119 (LOCKED A1): the citation surface is the READ docs only + # — this turn read nothing, so ZERO chips; the five suggested docs + # (ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > + # backups.md > kubernetes.md — the phase-119 D1 re-rank, pinned in + # the durable record below) seeded the prompt but never chip — + # the retired phase-118 A4 union is gone. The suggested tier itself + # is pinned below by the durable record (LOCKED A3, untouched). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # The rank-6+ remainder rides the related row: a labeled, # de-emphasized list — one .related-doc link per doc (rank order, @@ -162,13 +172,19 @@ def test_single_source_question_shows_exactly_one_citation_chip( row_log = db.scalars(select(QueryLog)).one() assert row_log.question == SINGLE_SOURCE_QUESTION assert row_log.deflected is False - # Suggested tier + related remainder, in the logged order. + # Suggested tier + related remainder, in the logged order. Phase + # 119 (D1): the stem sub-component ``aliases`` name-hits + # ``ssh_aliases.txt`` — its name-hit row leads the lexical list, + # the FTS rank shift re-orders the fused scores, and ``backups.md`` + # (best eff 0.016277) now out-ranks ``kubernetes.md`` (0.016036); + # re-pinned against the post-phase walk (the record's CONTENT — + # suggested + related + read, deduped — is LOCKED A3, untouched). assert row_log.sources == ( "docs/homelab/ssh/ssh_aliases.txt, " "docs/homelab/container_gitlab/gitlab-compose.yaml, " "docs/homelab/scripts/uptime_probe.py, " - "docs/homelab/kubernetes.md, " "docs/homelab/backups.md, " + "docs/homelab/kubernetes.md, " "docs/homelab/quadlet/compose.container, " "docs/homelab/networking/static-dns.json" ), row_log.sources diff --git a/tests/e2e/test_sources_midstream_bug.py b/tests/e2e/test_sources_midstream_bug.py index cd7fa92..92caab8 100644 --- a/tests/e2e/test_sources_midstream_bug.py +++ b/tests/e2e/test_sources_midstream_bug.py @@ -89,10 +89,6 @@ HESITATE_QUESTION = ( #: pre-content pause (SLOW_PRETOKEN_TRIGGER) is running. THINKING_TAIL = "nothing is invented" -#: Phase-10 viewer URL + phase-13 back=/ (byte-identical to the chip the -#: persistence suite pins — grounded-turn sources are unchanged by 20). -CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F" - async def _import_fixtures(mock_port: int) -> ImportSummary: kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"} @@ -334,7 +330,10 @@ def test_full_answer_completes_after_rag_nav_midstream( brain = msgs[1] assert brain["text"] == FULL_LONG assert brain["deflected"] is False - assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"]) + # Phase 119 (LOCKED A1): the turn read nothing, so its done sources + # — and the persisted record — are EMPTY (the retired phase-118 A4 + # union is gone). + assert brain["sources"] == [] # The turn SETTLED — the phase-48 query_log row exists (a # cancelled turn would leave no row at all). @@ -408,7 +407,11 @@ def test_nav_switch_before_first_token_completes( assert [m["who"] for m in msgs] == ["user", "brain"] assert MOCK_ANSWER_MARKER in msgs[1]["text"] assert msgs[1]["deflected"] is False - assert any(s["path"] == "homelab/kubernetes.md" for s in msgs[1]["sources"]) + # Phase 119 (LOCKED A1): the turn read nothing, so its done sources + # — the stored record's sources list — are EMPTY (the retired + # phase-118 A4 suggested-citation is gone; the full retrieval + # stays in query_log, pinned below). + assert msgs[1]["sources"] == [] # The turn settled — one finalized row (a cancelled turn would # leave no row at all). @@ -495,22 +498,25 @@ def test_completed_turn_unaffected( brain = before["messages"][1] assert MOCK_ANSWER_MARKER in brain["text"] assert brain["deflected"] is False - assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"]) + # Phase 119 (LOCKED A1): the turn read nothing, so its done sources + # — and the persisted record — are EMPTY (the retired phase-118 A4 + # union is gone). + assert brain["sources"] == [] # A trip to Sources and back (the turn finished long ago — uiState is # idle, so the pagehide save point must be a no-op). page.goto(app_url + "/sources.html") page.goto(app_url + "/") - # Full answer + source chip rendered; no error banner. + # Full answer rendered with ZERO citation chips (the zero-read turn + # persisted an empty sources list — phase 119, LOCKED A1; the + # retired phase-118 A4 chip pin is gone); no error banner. expect(page.locator("#empty-state")).to_be_hidden() expect(page.locator(".msg.user .bubble")).to_contain_text(QUESTION) bubble = page.locator(".msg.brain .bubble") expect(bubble).to_have_count(1) expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER) - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip).to_have_count(1) - expect(chip.first).to_have_attribute("href", CHIP_HREF) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) _no_error_banner(page) # Storage is byte-identical to the pre-navigation payload — the diff --git a/tests/e2e/test_summary_seed_context.py b/tests/e2e/test_summary_seed_context.py index f933064..3c4a6cc 100644 --- a/tests/e2e/test_summary_seed_context.py +++ b/tests/e2e/test_summary_seed_context.py @@ -59,19 +59,21 @@ TODO item — one Playwright file per story, A16): The inverse of the retired phase-24 pin: NO document's tail sentinel is in the echoed context — the full content of no suggested doc ever reached the model (case b). The grounded turn - cites the 5 suggested docs (case e's chip surface, no read yet) and + cites NOTHING (phase 119, LOCKED A1 — a zero-read turn chips + nothing; the retired phase-118 A4 suggested+read union is gone) and renders the de-emphasized related row (rank 6+); the durable - record carries suggested + related. + record carries suggested + related (118-A3 untouched). 2. ``test_read_suggested_doc_adds_full_text_and_cites`` — cases (c) + (e): the scripted flow ``read``s the rank-1 suggested doc — the read SUCCEEDS (phase 118: the seeds are summaries, not full text — the retired top-2 seed-read refusal is gone) and the mock's verbatim echo of the read result lands the tail sentinel in the answer (case c: the full text now arrives through the ``read`` - tool, not the seed); the citation chips = suggested + read, - deduped — the read doc is among the suggested, so the chip row is - exactly the 5 suggested docs, and the related row renders rank 6+ - (case e); the durable record carries suggested + related + read. + tool, not the seed); the citation chips = the READ docs only + (phase 119, LOCKED A1 — exactly the one read doc; none of the + four other suggested docs chips), and the related row renders + rank 6+ (case e); the durable record carries suggested + related + + read (118-A3 untouched). """ from __future__ import annotations @@ -503,14 +505,16 @@ def _last_query_log() -> QueryLog: return rows[0] -def _assert_suggested_chips_and_related_row(page: Page, suggested_paths: list[str]) -> None: - """The citation surface of a grounded turn (LOCKED A4): the chip row - is the suggested set (+ agent reads, deduped — asserted per test) - and the de-emphasized ``related-docs`` row carries the rank-6+ - remainder (phase-113 UI, never a citation chip).""" +def _assert_read_chips_and_related_row(page: Page, read_paths: list[str]) -> None: + """The citation surface of a grounded turn (phase 119, LOCKED A1 — + the retired phase-118 A4 union is gone): the chip row is the + AGENT-READ docs only (deduped, read order — a zero-read turn chips + NOTHING: the suggested docs are seed context, not citations), and + the de-emphasized ``related-docs`` row carries the rank-6+ remainder + (phase-113 UI, untouched — never a citation chip).""" chips = page.locator(".msg.brain .source-chip") - expect(chips).to_have_count(len(suggested_paths)) - for path in suggested_paths: + expect(chips).to_have_count(len(read_paths)) + for path in read_paths: expect( chips.filter(has_text=path), message=f"chip for {path}" ).to_have_count(1) @@ -553,11 +557,12 @@ def test_summaries_seed_the_prompt_not_the_full_text( assert _tool_frames(frames) == [], _tool_frames(frames) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False, done - # The grounded turn cites the 5 suggested docs (LOCKED A4 — no read - # yet: the chip row IS the suggested set, in rank order). - assert [ - (s["source"], s["path"]) for s in done["sources"] - ] == [(SOURCE, p) for p in SUGGESTED_TAIL], done["sources"] + # Phase 119 (LOCKED A1): the grounded done frame cites the READ docs + # only — this turn read NOTHING (the summary-only fast path), so the + # citation surface is empty; the suggested set is seed context, not + # citations (the retired phase-118 A4 union is gone). The durable + # record below still carries suggested + related (118-A3 untouched). + assert done["sources"] == [], done["sources"] bubble = _last_brain(page).locator(".bubble") expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) @@ -583,12 +588,13 @@ def test_summaries_seed_the_prompt_not_the_full_text( expect(bubble).not_to_contain_text(sentinel) expect(bubble).not_to_contain_text(TRUNCATION_MARKER) - # Case (e)'s chip surface (no read yet): chips = the 5 suggested - # docs; the related row renders rank 6+ (the de-emphasized row). - _assert_suggested_chips_and_related_row(page, SUGGESTED_TAIL) + # Case (e)'s chip surface (phase 119 A1 — no read yet): ZERO chips; + # the related row renders rank 6+ (the de-emphasized row, untouched). + _assert_read_chips_and_related_row(page, []) # Durable record: grounded; suggested + related (LOCKED A3 — the log - # records retrieval, not citations). + # records retrieval, not citations; phase 119 A1 retires the chip + # union only, not the record). row = _last_query_log() assert row.question == TAIL_QUESTION assert row.deflected is False @@ -609,10 +615,10 @@ def test_read_suggested_doc_adds_full_text_and_cites( retired top-2 seed-read refusal is gone) and the mock's verbatim echo of the read result lands the document's tail sentinel in the answer (case c: the full text reached the model through the - ``read`` tool, not the seed). The citation chips = suggested + - read, deduped — the read doc is among the suggested, so the chip - row is exactly the 5 suggested docs; the related row renders rank - 6+ (case e).""" + ``read`` tool, not the seed). The citation chips = the READ docs + only (phase 119, LOCKED A1 — exactly the one read doc; none of the + other four suggested docs chips; the retired phase-118 A4 union is + gone); the related row renders rank 6+ (case e, untouched).""" page.set_default_timeout(30_000) login(page, app_url, next="/") _install_page_hooks(page) @@ -628,13 +634,15 @@ def test_read_suggested_doc_adds_full_text_and_cites( ], _tool_frames(frames) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False, done - # LOCKED A4: done.sources = suggested + read, deduped — the read doc - # is already among the suggested, so the citation surface is - # exactly the 5 suggested docs (in rank order). + # Phase 119 (LOCKED A1): done.sources = the READ docs only — + # exactly the one scripted read; the other four suggested docs + # never chip (the retired phase-118 A4 suggested+read union is + # gone). assert [ (s["source"], s["path"]) for s in done["sources"] - ] == [(SOURCE, p) for p in SUGGESTED_READ], done["sources"] - # done.related = the rank-6+ remainder (deduped against the cited). + ] == [(SOURCE, READ_TARGET)], done["sources"] + # done.related = the rank-6+ remainder (deduped against the cited — + # the read doc is rank 1, not in the remainder). assert [ (s["source"], s["path"]) for s in done["related"] ] == [(SOURCE, p) for p in RELATED_PATHS], done["related"] @@ -658,13 +666,13 @@ def test_read_suggested_doc_adds_full_text_and_cites( if path != READ_TARGET: expect(bubble).not_to_contain_text(sentinel) - # Case (e): the UI chip row = suggested + read (deduped — the read - # doc is among the suggested, so chips = the 5 suggested docs), and - # the related row renders rank 6+. - _assert_suggested_chips_and_related_row(page, SUGGESTED_READ) + # Case (e): the UI chip row = the READ docs only (phase 119 A1 — + # exactly the one read doc), and the related row renders rank 6+. + _assert_read_chips_and_related_row(page, [READ_TARGET]) # Durable record: grounded; suggested + related + read (deduped, - # LOCKED A3). + # LOCKED A3 — phase 119 A1 retires the chip union only, not the + # record). row = _last_query_log() assert row.question == READ_QUESTION assert row.deflected is False diff --git a/tests/e2e/test_thinking_display.py b/tests/e2e/test_thinking_display.py index 580dba0..2363cc4 100644 --- a/tests/e2e/test_thinking_display.py +++ b/tests/e2e/test_thinking_display.py @@ -55,9 +55,6 @@ DEFLECT_PHRASE = r"haven't done anything like that" #: suite keys off it (mock_llm.compose_thinking). THINKING_FRAGMENT = "Step 2: Check my notes" STORAGE_KEY = "bor.chat.v1" -#: Phase-10 viewer URL + phase-13 back=/ (byte-identical to the chip the -#: persistence suite pins — grounded-turn sources are unchanged by 17). -CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F" async def _import_fixtures(mock_port: int) -> ImportSummary: @@ -154,13 +151,13 @@ def test_thinking_block_streams_open_then_collapses( expect(bubble).not_to_have_text("", timeout=30_000) expect(details).not_to_have_attribute("open") - # Settled: full scratchpad, grounded mock answer, source chip(s), - # and the re-enabled send button. + # Settled: full scratchpad, grounded mock answer, ZERO citation + # chips (phase 119, LOCKED A1 — the zero-read turn chips nothing; + # the retired phase-118 A4 union is gone), and the re-enabled send + # button. expect(details.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT) expect(bubble).to_contain_text(MOCK_ANSWER_MARKER) - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip.first).to_be_visible() - expect(chip.first).to_have_attribute("href", CHIP_HREF) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) expect(page.locator("#send-btn")).to_be_enabled() expect(page.locator("#send-label")).to_have_text("Send") @@ -232,10 +229,12 @@ def test_thinking_restored_after_reload(page: Page, app_url: str, seeded_kb: Non expect(restored).not_to_have_attribute("open") # restored COLLAPSED expect(restored.locator(".thinking-text")).to_have_text(captured) - # Answer bubble + source chip are intact (phase-14 restore path). + # Answer bubble is intact after the restore (phase-14 restore path) + # — with ZERO citation chips (phase 119, LOCKED A1: the zero-read + # turn persisted an empty sources list; the retired phase-118 A4 + # union is gone). expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER) - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip.first).to_have_attribute("href", CHIP_HREF) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # --------------------------------------------------------------------------- @@ -252,10 +251,10 @@ def test_no_thinking_block_without_trigger(page: Page, app_url: str, seeded_kb: # No trigger → no thinking events → no block anywhere on the page. expect(page.locator("details.thinking")).to_have_count(0) - # The turn itself is complete and grounded, exactly as before phase 17. - chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md") - expect(chip.first).to_be_visible() - expect(chip.first).to_have_attribute("href", CHIP_HREF) + # The turn itself is complete and grounded, exactly as before phase + # 17 — with ZERO citation chips (phase 119, LOCKED A1: the zero-read + # turn chips nothing; the retired phase-118 A4 union is gone). + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) # --------------------------------------------------------------------------- diff --git a/tests/fixtures/namekb/deploy/install-guide-a.md b/tests/fixtures/namekb/deploy/install-guide-a.md new file mode 100644 index 0000000..5b95aa5 --- /dev/null +++ b/tests/fixtures/namekb/deploy/install-guide-a.md @@ -0,0 +1,9 @@ +# Install Forgejo Guide A + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + +install forgejo with the deployment steps install forgejo with the deployment steps install forgejo with the deployment steps install forgejo with the deployment steps + +alpha1 alpha2 alpha3 alpha4 alpha5 alpha6 + +NAMEKB-TAIL-c5d6 diff --git a/tests/fixtures/namekb/deploy/install-guide-b.md b/tests/fixtures/namekb/deploy/install-guide-b.md new file mode 100644 index 0000000..578fd72 --- /dev/null +++ b/tests/fixtures/namekb/deploy/install-guide-b.md @@ -0,0 +1,9 @@ +# Install Forgejo Guide B + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + +install forgejo with the deployment steps install forgejo with the deployment steps install forgejo with the deployment steps + +beta1 beta2 beta3 beta4 beta5 beta6 + +NAMEKB-TAIL-d7e8 diff --git a/tests/fixtures/namekb/deploy/install-guide-c.md b/tests/fixtures/namekb/deploy/install-guide-c.md new file mode 100644 index 0000000..221f51e --- /dev/null +++ b/tests/fixtures/namekb/deploy/install-guide-c.md @@ -0,0 +1,9 @@ +# Install Forgejo Guide C + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + +install forgejo with the deployment steps install forgejo with the deployment steps + +gamma1 gamma2 gamma3 gamma4 gamma5 gamma6 + +NAMEKB-TAIL-f9a0 diff --git a/tests/fixtures/namekb/deploy/install-guide-d.md b/tests/fixtures/namekb/deploy/install-guide-d.md new file mode 100644 index 0000000..5e1fd3e --- /dev/null +++ b/tests/fixtures/namekb/deploy/install-guide-d.md @@ -0,0 +1,9 @@ +# Install Forgejo Guide D + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + +install forgejo with the deployment steps + +delta1 delta2 delta3 delta4 delta5 delta6 + +NAMEKB-TAIL-b1c2 diff --git a/tests/fixtures/namekb/forgejo/forgejo-home.md b/tests/fixtures/namekb/forgejo/forgejo-home.md new file mode 100644 index 0000000..cba51ca --- /dev/null +++ b/tests/fixtures/namekb/forgejo/forgejo-home.md @@ -0,0 +1,9 @@ +# Zeta Forge One + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + + + +kilo1 kilo2 kilo3 kilo4 kilo5 kilo6 + +NAMEKB-TAIL-a1b2 diff --git a/tests/fixtures/namekb/forgejo/forgejo-nginx.md b/tests/fixtures/namekb/forgejo/forgejo-nginx.md new file mode 100644 index 0000000..c633cfd --- /dev/null +++ b/tests/fixtures/namekb/forgejo/forgejo-nginx.md @@ -0,0 +1,9 @@ +# Zeta Forge Two + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + + + +lambda1 lambda2 lambda3 lambda4 lambda5 lambda6 + +NAMEKB-TAIL-b3c4 diff --git a/tests/fixtures/namekb/zz-folder-map.md b/tests/fixtures/namekb/zz-folder-map.md new file mode 100644 index 0000000..e24cc52 --- /dev/null +++ b/tests/fixtures/namekb/zz-folder-map.md @@ -0,0 +1,9 @@ +# Zeta Misc Three + +fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen + + + +micro1 micro2 micro3 micro4 micro5 micro6 + +NAMEKB-TAIL-e3f4 diff --git a/tests/fixtures/retrieval_battery.txt b/tests/fixtures/retrieval_battery.txt new file mode 100644 index 0000000..99c5fc3 --- /dev/null +++ b/tests/fixtures/retrieval_battery.txt @@ -0,0 +1,17 @@ +# Phase 119 retrieval battery (task 06, D6) — the 1,000-doc live replica. +# +# Gate (per the phase overview, post-phase measured numbers): +# 1-3. each name question's suggested top-5 contains its canonical doc: +# gitea -> deploy/Deployments/reeseapps/gitea/README.md (rank 4) +# forgejo-> 5/5 forgejo-tree docs, README #1 +# gateway-> deploy/Deployments/reeseapps/gateway/README.md (rank 3) +# 4. the 2026-09-05 incident regression: a qwen3.8-27b quadlet stays in +# the suggested top-5 (the digit-bearing prefix rule, unchanged). +# 5. deflection sanity: the out-of-KB question's gate verdict + fts +# count match the pre-phase replica run (HIGH, fts_hits=5) — the +# name-hit change must not move it. +How do I deploy gitea? +How do I deploy forgejo? +How do I set up the gateway? +Do you have notes on the qwen 3.8 27b model? +What is the capital of Mongolia? diff --git a/tests/integration/test_auth_api.py b/tests/integration/test_auth_api.py index f22ed32..2b6dde7 100644 --- a/tests/integration/test_auth_api.py +++ b/tests/integration/test_auth_api.py @@ -288,7 +288,10 @@ def test_token_user_surface_matrix(client: TestClient, db, seeded_kb: FakeRagLLM assert content_type.startswith("text/event-stream") assert frames[-1]["type"] == "done" assert frames[-1]["deflected"] is False - assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md" + # Phase 119 (A1): the citation surface is the agent's READ docs + # only — the canned turn reads nothing ⇒ no chips (the cited + # document's content pin below is the surface's other half). + assert frames[-1]["sources"] == [] # Suggestion chips + the cited document's content (the viewer). assert client.get("/api/suggestions").status_code == 200 @@ -455,7 +458,8 @@ def test_admin_chat_still_streams(client: TestClient, db, seeded_kb: FakeRagLLM) assert len(deltas) >= 2 # genuinely streamed assert frames[-1]["type"] == "done" assert frames[-1]["deflected"] is False - assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md" + # Phase 119 (A1): read docs only — nothing was read ⇒ no chips. + assert frames[-1]["sources"] == [] # ---------- phase 81: rate-limited failed sign-ins (audit SEC-03) ---------- diff --git a/tests/integration/test_chat_api.py b/tests/integration/test_chat_api.py index 8a1c01d..8f05b64 100644 --- a/tests/integration/test_chat_api.py +++ b/tests/integration/test_chat_api.py @@ -280,14 +280,14 @@ def test_chat_streams_deltas_then_done_with_sources(client, db, seeded_kb: FakeR assert frames[-1]["type"] == "done" # done is the final event assert done[0]["deflected"] is False assert done[0]["suggestions"] == [] - sources = done[0]["sources"] - assert sources, "done must carry the cited sources" - # Phase 118 (A4): the citation surface is the suggested tier (top-5, - # no floor) + the agent's reads (none on this turn) — deduped. - assert len(sources) == get_settings().suggested_docs - assert sources[0]["path"] == "homelab/kubernetes.md" - assert sources[0]["source"] == "docs" - assert sources[0]["title"] == "Kubernetes Homelab Cluster" + # Phase 119 (LOCKED A1 — the phase-118 A4 union retired): the + # citation surface is the agent's READ docs only — this turn's + # canned LLM never emits a tool call, so nothing was read and the + # grounded done frame chips nothing (an accepted, owner-directed + # consequence — the answer prose names the doc it used). The + # retrieval stays durably recorded (118-A3; pinned in the + # query_log test below this one). + assert done[0]["sources"] == [] # The LLM received the locked HIGH prompt — the ```` block # seeds the document's stored SUMMARY (phase 118, LOCKED A6: summary @@ -334,13 +334,13 @@ def test_chat_streams_thinking_before_deltas(client, db, seeded_kb: FakeRagLLM) assert "".join(f["text"] for f in thinking) == thinker.thinking assert "".join(d["text"] for d in deltas) == thinker.answer - # Done still last; sources unchanged by the thinking extension. + # Done still last; the citation surface is unchanged by the thinking + # extension — phase 119 (A1): read docs only, nothing read ⇒ none. done = frames[-1] assert done["type"] == "done" assert done["deflected"] is False assert done["suggestions"] == [] - assert done["sources"][0]["path"] == "homelab/kubernetes.md" - assert done["sources"][0]["source"] == "docs" + assert done["sources"] == [] assert not any(f.get("type") == "error" for f in frames) @@ -469,12 +469,13 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM) def test_done_frame_carries_related_tier_on_grounded_turn( client, db, seeded_kb: FakeRagLLM ) -> None: - """Phase 118 (LOCKED A3/A4): a grounded turn's done frame carries the - suggested tier in ``sources`` (top-5, no floor) and the related - tier — the ranked docs from rank 6+ after the suggested set, capped - at ``related_max_docs`` (2) — in ``related``, disjoint from the - citation surface. The durable record keeps the FULL retrieval - (suggested + related + read, LOCKED A3).""" + """Phase 118 (A3) × phase 119 (A1): a grounded turn's done frame + carries the related tier — the ranked docs from rank 6+ after the + suggested set, capped at ``related_max_docs`` (2) — in ``related``, + disjoint from the read-only ``sources`` (nothing was read on this + turn ⇒ no chips; the never-read suggested tier is not on the wire). + The durable record keeps the FULL retrieval (suggested + related + + read, 118-A3 stands).""" fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb try: _, _, frames = _stream_chat(client, QUESTION) @@ -483,11 +484,11 @@ def test_done_frame_carries_related_tier_on_grounded_turn( done = frames[-1] assert done["deflected"] is False - # A4: the citation surface is the suggested tier (5, no read on this - # turn) — ranks 1–5 for the Kubernetes question. + # Phase 119 (A1): the citation surface is the agent's READ docs + # only — no read on this turn ⇒ no chips (the retired A4 union + # would have carried the suggested ranks 1–5 here). sources = [(s["source"], s["path"]) for s in done["sources"]] - assert len(sources) == get_settings().suggested_docs - assert sources[0] == ("docs", "homelab/kubernetes.md") + assert sources == [] related = done["related"] # Rank 6–7 for the Kubernetes question (after the top-5 suggested # set), capped at related_max_docs. @@ -546,12 +547,13 @@ def test_deflected_done_frame_carries_weak_hits_in_related( def test_related_doc_read_by_agent_is_cited_not_related( client, db, seeded_kb: FakeRagLLM ) -> None: - """Phase 118 × phase 37: an agent-read doc is a citation by definition - (LOCKED A4) — when the agent ``read``s a rank-6+ doc (the related - tier, "nearby docs"), it joins done.sources (deduped, last — it was - not suggested, so the read appends it) and is EXCLUDED from - done.related (a "nearby doc" that was actually used must not read as - nearby).""" + """Phase 119 (A1) × phase 37: an agent-read doc is a chip by + definition — when the agent ``read``s a rank-6+ doc (the related + tier, "nearby docs"), it is the done frame's ONLY chip (the read + docs ARE the citation surface since phase 119; the never-read + suggested tier is not on the wire) and is EXCLUDED from + done.related (a "nearby doc" that was actually used must not read + as nearby — unchanged intent, the dedupe keyed on read docs).""" scripted = FakeRagLLM( tool_script=[ [ @@ -572,9 +574,9 @@ def test_related_doc_read_by_agent_is_cited_not_related( done = frames[-1] assert done["deflected"] is False sources = [(s["source"], s["path"]) for s in done["sources"]] - # A4: suggested (5) + the read doc (last). - assert len(sources) == get_settings().suggested_docs + 1 - assert sources[-1] == ("docs", "homelab/quadlet/cache.volume") # read ⇒ cited + # A1: read docs only — the read related doc is the ONLY chip (the + # retired A4 union would have carried 6 here). + assert sources == [("docs", "homelab/quadlet/cache.volume")] # read ⇒ chip related = [(s["source"], s["path"]) for s in done["related"]] assert ("docs", "homelab/quadlet/cache.volume") not in related assert set(sources).isdisjoint(set(related)) @@ -605,8 +607,9 @@ def test_keyword_question_grounded_by_lexical_hit_despite_weak_cosine( assert done["type"] == "done" assert done["deflected"] is False # weak cosine, but a lexical hit assert done["suggestions"] == [] - sources = done["sources"] - assert sources and sources[0]["path"] == "homelab/networking/static-dns.json" + # Phase 119 (A1): read docs only — nothing was read ⇒ no chips; + # the lexical hit stays in the durable record (asserted below). + assert done["sources"] == [] (system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1] assert "HIGH" in system["content"] # grounded prompt @@ -1085,12 +1088,13 @@ def test_truncated_read_streams_tool_result_frame_after_tool_frame( assert body.startswith(f"Document docs/big-read.md:\ndate: 2024-06-15\n{content[:cap]}") assert TRUNCATION_MARKER in body assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body - # The truncated read is still a SUCCESSFUL call — cited in done. + # The truncated read is still a SUCCESSFUL call — it is the frame's + # only chip (phase 119, A1: read docs only; the suggested kubernetes + # doc was never read, so it is not on the wire). done = frames[-1] assert done["type"] == "done" and done["deflected"] is False - assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]] - assert ("docs", "homelab/kubernetes.md") in [ - (s["source"], s["path"]) for s in done["sources"] + assert [(s["source"], s["path"]) for s in done["sources"]] == [ + ("docs", "big-read.md") ] @@ -1159,11 +1163,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc( client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture ) -> None: """(a) Grounded turn with tool calls: the event sequence is - ``thinking?/tool/tool/delta…/done``; ``done.sources`` and the - ``query_log`` row include the read document (deduped, order - preserved); the per-turn log line carries ``tool_calls=2``. - Phase 45: the agent loop keeps offering the tools for the whole - turn — the round cap (not per-tool budgets) is the bound.""" + ``thinking?/tool/tool/delta…/done``; ``done.sources`` is the read + document (phase 119, A1 — the read docs are the citation surface) + and the ``query_log`` row includes it (deduped, order preserved); + the per-turn log line carries ``tool_calls=2``. Phase 45: the + agent loop keeps offering the tools for the whole turn — the round + cap (not per-tool budgets) is the bound.""" scripted = FakeRagLLM( tool_script=[ [ @@ -1212,12 +1217,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc( done = frames[-1] assert done["deflected"] is False - # done.sources = the retrieval docs + the read doc, deduped, order kept. + # Phase 119 (A1): done.sources = the agent's READ docs only — the + # read doc is the frame's only chip (the suggested retrieval docs + # are not on the wire; they stay in the durable record below). sources = [(s["source"], s["path"]) for s in done["sources"]] - assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited - assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs - assert len(sources) == len(set(sources)) # deduped by (source, path) - assert done["sources"][-1]["title"] == "Backup Strategy" + assert sources == [("docs", "homelab/backups.md")] # the read doc is cited + assert done["sources"][0]["title"] == "Backup Strategy" # Phase 45: the tools stay offered on every request — the round cap # (not spent budgets) bounds the loop, and the model answered while @@ -1250,7 +1255,8 @@ def test_grounded_turn_streams_grep_tool_frames( the raw pattern is the frame's ``argument`` (the UI renders the "searching for" line from it). A non-string pattern — a model error the backend refuses — yields ``argument: null``. A grep adds no - source: ``done.sources`` stays the retrieval docs (locked A5).""" + source (locked A5): with no read on the turn, ``done.sources`` is + empty (phase 119, A1).""" scripted = FakeRagLLM( tool_script=[ [ @@ -1292,9 +1298,8 @@ def test_grounded_turn_streams_grep_tool_frames( assert [f for f in frames if f["type"] == "delta"] done = frames[-1] assert done["type"] == "done" and done["deflected"] is False - paths = [s["path"] for s in done["sources"]] - assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged - assert "homelab/backups.md" not in paths # a grep adds no source + # A grep adds no source (A5) and nothing was read (A1) ⇒ no chips. + assert done["sources"] == [] def test_tool_frames_carry_the_model_arguments_regardless_of_execution( @@ -1306,7 +1311,8 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution( and frame emission is execution-independent: a rejected call (an unknown ``read`` path) still streams its frame with the model's argument as-is. The rejected read adds no source (``done.sources`` - stays the retrieval docs), and rejected calls count nothing + carries no chip — the read failed and nothing else was read, + phase 119 A1), and rejected calls count nothing (``tool_calls=1`` — only the executed scoped ``ls``).""" # The scoped ``ls`` source-name check reads the registry — insert a # row resolving to ``docs`` (the fixture's source name) and delete @@ -1350,12 +1356,12 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution( # passed — frame emission is execution-independent. assert read_frame["argument"] == "docs/homelab/nope.md" - # The rejected read adds no source — done.sources stays retrieval. + # The rejected read adds no source — and the read-only surface is + # empty on this turn (nothing was read; the refused read cites + # nothing, phase 119 A1). done = frames[-1] assert done["type"] == "done" and done["deflected"] is False - paths = [s["path"] for s in done["sources"]] - assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged - assert "homelab/nope.md" not in paths # the refused read cites nothing + assert done["sources"] == [] # The rejected call counts nothing — only the executed scoped ls. lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()] @@ -1462,9 +1468,10 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request( done = frames[-1] assert done["type"] == "done" assert done["deflected"] is False - paths = [s["path"] for s in done["sources"]] - assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged - assert "homelab/backups.md" not in paths # nothing was read + # Phase 119 (A1): nothing was read ⇒ no chips (the kill-switch + # turn's citation surface is empty; the retrieval stays in the + # durable record below). + assert done["sources"] == [] # Exactly one request, and it carried no ``tools`` key at all — the # scripted tool calls were never even offered a chance. diff --git a/tests/integration/test_kb_overview_api.py b/tests/integration/test_kb_overview_api.py index feff291..58068db 100644 --- a/tests/integration/test_kb_overview_api.py +++ b/tests/integration/test_kb_overview_api.py @@ -25,15 +25,17 @@ from pathlib import Path import pytest from fastapi.testclient import TestClient -from sqlalchemy import select, text +from sqlalchemy import text from test_chat_api import FakeRagLLM, _stream_chat, _token_vec from app.api import chat as chat_api +from app.config import get_settings from app.main import app as fastapi_app from app.models import Document, KbOverview +from app.rag.agent import suggested_folder_lines from app.rag.importer import import_sources from app.rag.prompts import build_deflect_prompt, build_high_prompt -from app.rag.retriever import retrieve, weak_hit_titles +from app.rag.retriever import retrieve, select_suggested, weak_hit_titles from tests.conftest import ADMIN_PASSWORD FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs" @@ -90,15 +92,27 @@ def _seed_overview(db) -> None: db.commit() -def _cited_docs(db, frames: list[dict]) -> list[Document]: - """The documents the done event cited, in citation order — the same - list ``plan_turn`` passed to the prompt builder.""" - docs = [] - for s in frames[-1]["sources"]: - doc = db.scalar(select(Document).where(Document.path == s["path"])) - assert doc is not None, f"done source {s['path']!r} missing from the KB" - docs.append(doc) - return docs +def _suggested_docs(db) -> list[Document]: + """The documents the grounded turn's HIGH prompt seeded, in the same + order ``plan_turn`` walked them — the SAME deterministic suggested + walk over the retrieval. Phase 119 (A1) moved the citation surface + to the agent's READ docs, so the seeded tier can no longer be read + off the done frame (the canned LLM here reads nothing ⇒ + ``done.sources`` is empty) — reconstructing from the retrieval is + the faithful source now.""" + chunks = retrieve(db, QUESTION, _token_vec(QUESTION)) + return list(select_suggested(chunks, n=get_settings().suggested_docs)) + + +def _suggested_folder_lines(db) -> list[str]: + """The suggested-folder context lines the endpoint computed (phase + 119, D3, LOCKED A4) — the SAME deterministic suggested walk + ``plan_turn`` performs internally (one extra walk, no shared + state), over the real seeded catalog.""" + chunks = retrieve(db, QUESTION, _token_vec(QUESTION)) + return suggested_folder_lines( + db, select_suggested(chunks, n=get_settings().suggested_docs) + ) def _turn_log_lines(caplog: pytest.LogCaptureFixture) -> list[str]: @@ -124,7 +138,14 @@ def test_no_row_high_prompt_byte_identical_to_pre_phase( assert frames[-1]["deflected"] is False (system, user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1] assert user["content"] == QUESTION - expected = build_high_prompt(_cited_docs(db, frames), notes=[], kb_overview=None) + # Phase 119 (D3): the endpoint's suggested-folder lines ride the + # HIGH prompt — reconstructed the same deterministic way. + expected = build_high_prompt( + _suggested_docs(db), + notes=[], + kb_overview=None, + folder_lines=_suggested_folder_lines(db), + ) assert system["content"] == expected assert "" not in system["content"] @@ -179,8 +200,14 @@ def test_row_high_prompt_carries_kb_section_exactly( assert frames[-1]["deflected"] is False (system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1] + # Phase 119 (D3): the suggested-folder lines ride the HIGH prompt + # alongside the section — reconstructed the same + # deterministic way. expected = build_high_prompt( - _cited_docs(db, frames), notes=[], kb_overview=OVERVIEW + _suggested_docs(db), + notes=[], + kb_overview=OVERVIEW, + folder_lines=_suggested_folder_lines(db), ) assert system["content"] == expected diff --git a/tests/integration/test_name_hit_lexical.py b/tests/integration/test_name_hit_lexical.py index 844cdaa..a81add4 100644 --- a/tests/integration/test_name_hit_lexical.py +++ b/tests/integration/test_name_hit_lexical.py @@ -6,10 +6,12 @@ mapping with fake rows; this suite covers the SQL side on real Postgres: the document-projection scan, the LATERAL representative- chunk fetch (the ``is_summary`` chunk wins, chunk 0 otherwise, and a chunk-less name match is EXCLUDED — the ``c.id IS NOT NULL`` guard), -the (count, length, catalog) ranking, the name-hits-lead-the-lexical- -list union with the FTS rows (chunk-id dedup), and the full -``retrieve()`` → ``select_documents()`` path putting the versioned- -name document into the seeded top-N. +the (matched-token count, catalog) ranking (phase 119, LOCKED A2 — +the two-class component rule: digit-bearing prefix, digitless exact, +titles never matched), the name-hits-lead-the-lexical-list union with +the FTS rows (chunk-id dedup), and the full ``retrieve()`` → +``select_documents()`` path putting the versioned-name document into +the seeded top-N. Requires: ``podman compose up -d db``. """ @@ -137,19 +139,29 @@ def kb(db) -> Iterator[None]: def test_name_hit_chunks_real_sql(kb, db) -> None: - """Real Postgres: the projection scan finds exactly the qwen3.8 - quadlet (the qwen3.6 sibling and the chunk-less name match are - excluded), and the LATERAL fetch hands back the SUMMARY chunk as - the representative (position −1, is_summary).""" + """Real Postgres: the projection scan finds the qwen3.8 quadlet + (the ``qwen38`` stem prefix — the incident's original case) AND + the ``notes/llama.cpp.md`` doc (the question names "llama.cpp" — + the dotted token ``llamacpp`` exact-matches the file stem, the + phase-119 two-class rule). The qwen3.6 sibling (``qwen38`` is not + a prefix of ``qwen36…``) and the chunk-less name match are + excluded; the LATERAL fetch hands back the SUMMARY chunk as the + representative for the quadlet (position −1, is_summary) and chunk + 0 for the single-chunk notes doc.""" out = _name_hit_chunks(db, INCIDENT_QUESTION) assert [rc.document.path for rc in out] == [ - "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container" + "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container", + "notes/llama.cpp.md", # the question names it — stem exact (A2) ] rc = out[0] assert rc.position == -1 # the summary chunk wins the LATERAL order assert rc.is_summary is True assert rc.fts_hit is True # the lexical signal — the A8 gate answers assert rc.cosine == 0.0 # no vector rank on the name-hit row + rc_notes = out[1] + assert rc_notes.position == 0 # chunk 0 (no summary chunk) + assert rc_notes.is_summary is False + assert all(rc.name_hit is True for rc in out) # phase 119 — D2 bonus input assert "qwen3.8-empty.container" not in [r.document.path for r in out] # chunk-less guard @@ -163,13 +175,19 @@ def test_lexical_candidates_name_hit_leads_real_sql(kb, db) -> None: assert paths[0] == ( "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container" ) + assert paths[1] == "notes/llama.cpp.md" # the second name hit (stem exact) # The FTS pollution is still present (the incident's shape) — but - # behind the name hit, no longer ahead of it. + # behind the name hits, no longer ahead of them. assert ( "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container" in paths ) assert all(rc.fts_hit is True for rc in out) + # Phase 119: the two name-hit representative rows are flagged, the + # plain FTS rows are not. + assert out[0].name_hit is True + assert out[1].name_hit is True + assert all(not rc.name_hit for rc in out[2:]) def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None: @@ -177,7 +195,12 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None: fused) → ``select_documents`` puts the qwen3.8 quadlet in the seeded top-N — the incident's seed miss (the two overview docs only) is fixed. The question vector is parallel to the q38 chunk - embeddings (cosine 1.0), orthogonal to q36 (0.0).""" + embeddings (cosine 1.0), orthogonal to q36 (0.0). + + Phase 119: the ``name_hit`` flag survives the fusion — the two + name-hit representative chunks are double hits (vector ∪ lexical), + and the double-hit merge ORs the flag into the surviving row; + the plain q36 vector+FTS row stays False.""" question_vec = _vec(1) chunks = retrieve(db, INCIDENT_QUESTION, question_vec) docs = select_documents(chunks, n=2) @@ -185,16 +208,31 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None: "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container", "notes/llama.cpp.md", ] + by_path: dict[str, list[bool]] = {} + for rc in chunks: + by_path.setdefault(rc.document.path, []).append(rc.name_hit) + # The q38 doc's summary chunk is the double-hit name hit (flagged); + # its plain chunk 0 (vector + FTS) is not — the doc has a flagged row. + assert any(by_path[ + "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container" + ]) + assert any(by_path["notes/llama.cpp.md"]) # its only chunk is the name hit + assert not any(by_path[ + "reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container" + ]) # plain vector+FTS — never a name hit def test_name_hit_limit_real_sql(db) -> None: - """Twelve identical (1, 6) name hits — the LATERAL fetch (and the - output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog order.""" + """Twelve tied name hits (one matched token each — the ``qwen38`` + stem PREFIX; the token must lead the stem, the old mid-stem + containment no longer matches) — the LATERAL fetch (and the + output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog + order.""" db.execute(text("TRUNCATE chunks, documents")) db.commit() for i in range(12): doc = _doc( - db, "S", f"quadlets/m{i:02d}-qwen38.container", f"m{i:02d}-qwen38", "llama cpp qwen38\n" + db, "S", f"quadlets/qwen38-m{i:02d}.container", f"qwen38-m{i:02d}", "llama cpp qwen38\n" ) db.flush() c = _chunk(db, doc, 0, f"llama cpp qwen38 doc {i}") @@ -203,5 +241,6 @@ def test_name_hit_limit_real_sql(db) -> None: out = _name_hit_chunks(db, "what are the llama.cpp arguments for qwen 3.8") assert len(out) == NAME_HIT_LIMIT assert [rc.document.path for rc in out] == [ - f"quadlets/m{i:02d}-qwen38.container" for i in range(NAME_HIT_LIMIT) + f"quadlets/qwen38-m{i:02d}.container" for i in range(NAME_HIT_LIMIT) ] + assert all(rc.name_hit is True for rc in out) diff --git a/tests/integration/test_recency_boost.py b/tests/integration/test_recency_boost.py index ec176a4..1665c52 100644 --- a/tests/integration/test_recency_boost.py +++ b/tests/integration/test_recency_boost.py @@ -12,13 +12,21 @@ margins below are exact floats, not flaky measurements. **Measured geometry (recorded per task step 4/5):** * Owner scenario — A (``backups/retention.md``, created 2020-01-01, - the exact answer, cosine 1.0) lands at vector rank 1 + FTS rank 1 - (fused 0.03278689); B (``backups/retention-draft.md``, created + the exact answer, cosine 1.0) lands at vector rank 1 + lexical rank + 3 (fused 0.03226646); B (``backups/retention-draft.md``, created yesterday, the "under review, no decision yet" draft, cosine - 0.707107) lands at vector rank 10 + FTS rank 3 (fused 0.03015873 — - a solid FTS hit at rank 3, as the task describes). Pre-boost fused - margin **A−B = 0.00262816** (asserted ≥ 3× the zero-age boost = - 0.002100 at the default → ratio 1.25, the "comfortable margin"). + 0.707107) lands at vector rank 10 + lexical rank 1 (fused + 0.03067915). Pre-boost fused margin **A−B = 1/63 − 1/70 = + 0.00158730** — re-pinned by phase 119 (task 01): the two-class + name-hit rule (LOCKED A2) makes the digitless question token + "retention" exact-match the stem sub-component of all three + ``retention*`` docs, so the lexical list now LEADS with those name + hits in catalog order (draft, review, retention) and B takes the + rank-1 lexical slot (pre-119 it was a plain FTS rank-1 / rank-3 + hit). The flip-safety bar the margin is pinned against — margin > + the zero-age boost, so a zero-age B's full boost cannot close it + (A's own boost being ≈ 0 at its 2020 age) — still holds, at 2.27× + the default. * Twin near-tie — C (``twin/c-older.md``, 2019) and D (``twin/d-newer.md``, yesterday) with IDENTICAL chunk text and near-identical vectors (cosine 1.0 vs 0.9999 — a literal identical @@ -29,12 +37,16 @@ margins below are exact floats, not flaky measurements. scale. * The DEFAULT was tuned from the design starting point (0.001) down to **0.0007** (task step 5: "tune the DEFAULTS … until old-correct wins - comfortably"): on the k=60 scale the owner scenario's margin is + comfortably"): on the k=60 scale the owner scenario's margin was 0.00262816 < 3×0.001, and a 0.001 zero-age boost (+0.000997 for a yesterday doc) would have FLIPPED the pinned scenario. 0.0007 keeps the flip margin comfortable (0.000698 > 0.00052882, lead - +0.000169) while staying 1.25× under the 3×-boost margin bar. The - owner re-tunes live via ``BOR_RECENCY_BOOST``. + +0.000169). Phase 119 (task 01) re-pinned the owner margin to + 0.00158730 (above): still above the zero-age boost (2.27×), so the + worst-case flip — a zero-age B against A's ≈ 0 boost — keeps A + first; the twin-flip figures (gap 0.00052882, boost 0.000698) are + untouched (the twin KB has no name hits). The owner re-tunes live + via ``BOR_RECENCY_BOOST``. Requires: ``podman compose up -d db``. """ @@ -239,11 +251,12 @@ def test_owner_scenario_old_correct_beats_new_similar( owner_kb, db, monkeypatch: pytest.MonkeyPatch ) -> None: """THE owner scenario, pinned at the DEFAULTS: the older doc that - answers ranks above the newer similar one — AND the pre-boost fused - margin is ≥ 3× the zero-age boost (the "comfortable margin"; the - measured 0.00262816 vs the 0.0021 bar is recorded in the module - docstring). Re-pinned with the boost OFF: relevance alone already - ordered them (no regression — the boost is not what makes A win).""" + answers ranks above the newer similar one — AND the pre-boost + fused margin exceeds the zero-age boost (the flip-safety bar; the + phase-119-re-pinned measured 0.00158730 vs the 0.0007 default is + recorded in the module docstring). Re-pinned with the boost OFF: + relevance alone already ordered them (no regression — the boost is + not what makes A win).""" chunks = retrieve(db, QUESTION, QUESTION_VEC) assert select_documents(chunks, n=2)[0].path == "backups/retention.md" @@ -254,7 +267,15 @@ def test_owner_scenario_old_correct_beats_new_similar( fused = fuse(vector, lexical, s.rrf_k) by_path = {rc.document.path: rc.score for rc in fused} margin = by_path["backups/retention.md"] - by_path["backups/retention-draft.md"] - assert margin >= 3 * s.recency_boost + # Phase 119 (task 01) re-pinned geometry: the three ``retention*`` + # docs are name hits (the digitless token "retention" exact-matches + # the stem sub-component), ranked in catalog order (draft, review, + # retention) — so A sits at vector rank 1 + lexical rank 3 and B + # at vector rank 10 + lexical rank 1: the margin is exactly + # 1/63 − 1/70. The flip-safety bar: a zero-age B's full boost + # cannot close it (A's own boost is ≈ 0 at its 2020 age). + assert margin == pytest.approx(1 / 63 - 1 / 70) + assert margin > s.recency_boost # 2.27× at the default # The kill switch: A still first (relevance alone), and the # weight-0 scores are the pre-phase fused scores byte-identical. diff --git a/tests/unit/test_agent.py b/tests/unit/test_agent.py index cd3a4cf..bd2b3d5 100644 --- a/tests/unit/test_agent.py +++ b/tests/unit/test_agent.py @@ -1091,6 +1091,229 @@ def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None: assert total50 == 50 and len(files50) == 50 +# ---------- phase 119 (D3): the suggested-folder context lines ---------- + + +def _patch_catalog( + monkeypatch: pytest.MonkeyPatch, catalog: dict[str, list[tuple[str, str, str]]] +) -> None: + """Monkeypatch the ``ls`` fetchers (house style) with one source + catalog of ``(path, title, date)`` rows per source name.""" + monkeypatch.setattr( + agent, "_source_document_rows", lambda db, source: catalog.get(source, []) + ) + monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {}) + + +def test_suggested_folder_lines_source_root_doc_lists_the_top_level( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A suggested doc at the source root (``""`` prefix) renders the + ``/: …`` line with the source's top level — subfolders with + recursive counts, the suggested doc itself excluded.""" + _patch_catalog( + monkeypatch, + { + "Homelab": [ + ("a/b/one.md", "One", "2024-06-15"), + ("a/b/two.md", "Two", "2024-06-15"), + ("a/c.md", "C", "2024-06-15"), + ("z.md", "Z", "2024-06-15"), + ] + }, + ) + assert agent.suggested_folder_lines( + cast("Session", object()), [_doc("Homelab", "z.md")] + ) == ["Homelab/: a/ (3 docs)"] + + +def test_suggested_folder_lines_nested_doc_lists_the_parent_folder( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A nested suggested doc renders the PARENT folder's line — the + direct children only: subfolder names (source-relative, recursive + counts) + the files by relative filename, the suggested doc itself + out of the entries.""" + _patch_catalog( + monkeypatch, + { + "Homelab": [ + ("a/b/one.md", "One", "2024-06-15"), + ("a/b/two.md", "Two", "2024-06-15"), + ("a/c/d.md", "D", "2024-06-15"), + ("a/e.md", "E", "2024-06-15"), + ] + }, + ) + # The parent of a/b/one.md is a/b: one.md excluded, two.md stays. + assert agent.suggested_folder_lines( + cast("Session", object()), [_doc("Homelab", "a/b/one.md")] + ) == ["Homelab/a/b/: two.md"] + # The parent of a/e.md is a: the subfolders (recursive counts — a/b + # carries two, a/c one) first, then the files; e.md excluded, so no + # file entries remain. + assert agent.suggested_folder_lines( + cast("Session", object()), [_doc("Homelab", "a/e.md")] + ) == ["Homelab/a/: a/b/ (2 docs), a/c/ (1 doc)"] + + +def test_suggested_folder_lines_excludes_only_the_owning_doc( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The doc that OWNS the line (the first suggested doc in that + folder) is the only one excluded — a folder whose sole entry was + the suggested doc renders its header alone (the empty-level + precedent).""" + _patch_catalog( + monkeypatch, + { + "Homelab": [("a/only.md", "Only", "2024-06-15")] + }, + ) + assert agent.suggested_folder_lines( + cast("Session", object()), [_doc("Homelab", "a/only.md")] + ) == ["Homelab/a/:"] + + +def test_suggested_folder_lines_dedupes_by_source_and_prefix( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Dedupe: two suggested docs in the same folder ⇒ ONE line (the + first doc's position, the first doc excluded, the second listed); + the key is ``(source, prefix)`` — the same prefix under a DIFFERENT + source is a distinct line; suggested-doc order is the line order.""" + _patch_catalog( + monkeypatch, + { + "Homelab": [ + ("a/b/one.md", "One", "2024-06-15"), + ("a/b/two.md", "Two", "2024-06-15"), + ("a/c.md", "C", "2024-06-15"), + ("a/d/e.md", "E", "2024-06-15"), + ("z.md", "Z", "2024-06-15"), + ], + "Other": [("a/b/x.md", "X", "2024-06-15")], + }, + ) + docs = [ + _doc("Homelab", "z.md"), + _doc("Homelab", "a/b/one.md"), + _doc("Homelab", "a/b/two.md"), # deduped (a/b already seen) + _doc("Homelab", "a/c.md"), + _doc("Homelab", "a/d/e.md"), + _doc("Other", "a/b/x.md"), # same prefix, different source + ] + assert agent.suggested_folder_lines(cast("Session", object()), docs) == [ + "Homelab/: a/ (4 docs)", + "Homelab/a/b/: two.md", + "Homelab/a/: a/b/ (2 docs), a/d/ (1 doc)", + "Homelab/a/d/:", + "Other/a/b/:", + ] + + +def test_suggested_folder_lines_max_lines_caps_the_list( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """At most *max_lines* lines — the walk stops, suggested-doc order + kept (the first distinct folders win).""" + _patch_catalog( + monkeypatch, + { + "Homelab": [ + ("f1/x.md", "1", "2024-06-15"), + ("f2/x.md", "2", "2024-06-15"), + ("f3/x.md", "3", "2024-06-15"), + ] + }, + ) + docs = [_doc("Homelab", p) for p in ("f1/x.md", "f2/x.md", "f3/x.md")] + lines = agent.suggested_folder_lines(cast("Session", object()), docs, max_lines=2) + assert lines == ["Homelab/f1/:", "Homelab/f2/:"] + assert agent.suggested_folder_lines( + cast("Session", object()), docs, max_lines=1 + ) == ["Homelab/f1/:"] + + +def test_suggested_folder_lines_max_entries_cap_and_remainder( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """At most *max_entries* entries, then `` +N more`` with the EXACT + remainder (the true pre-cap folder total — the owning doc leaves + it); the ``(1 doc)`` singular form; no `` +N more`` when every + entry fits.""" + rows = [(f"big/s{i}/x.md", f"S{i}", "2024-06-15") for i in range(4)] + [ + (f"big/f{i:02d}.md", f"F{i}", "2024-06-15") for i in range(10) + ] + _patch_catalog(monkeypatch, {"S": rows}) + db = cast("Session", object()) + # 4 subfolders (one doc each) + 10 files − the owning f00.md = 13 + # entries; 8 shown (the 4 subfolders + f01..f04) + 5 more. The + # subfolder entries are source-relative (``big/s0/``) — exactly as + # the model's own ls output names them. + assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")]) == [ + "S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc), big/s3/ (1 doc), " + "f01.md, f02.md, f03.md, f04.md +5 more" + ] + # A wide enough cap shows every entry — no remainder marker. + assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")], max_entries=20) == [ + "S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc), big/s3/ (1 doc), " + "f01.md, f02.md, f03.md, f04.md, f05.md, f06.md, f07.md, f08.md, f09.md" + ] + # The remainder is exact at a tighter cap too (3 subfolders shown — + # files only follow ALL subfolders, the ls order — 10 left). + assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")], max_entries=3) == [ + "S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc) +10 more" + ] + + +def test_suggested_folder_lines_entry_order_matches_the_ls_listing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The line's entry order IS the ``ls`` folder-level rendering order + (pinned against :func:`group_folder_listing` itself): the direct + subfolders first (path order), then the files (path order) — a + mixed folder where the file names and the subfolder names interleave + in path order still lists all subfolders before any file.""" + rows = [ + ("a/a.md", "A", "2024-06-15"), + ("a/b.md", "B", "2024-06-15"), + ("a/c/d.md", "D", "2024-06-15"), + ] + _patch_catalog(monkeypatch, {"Homelab": rows}) + db = cast("Session", object()) + suggested = _doc("Homelab", "a/b.md") + # The expected entry order, derived from group_folder_listing itself + # (subfolders → files, minus the owning doc) — the line must read + # the same as the model's own ls output. + sub, files, total = agent.group_folder_listing("Homelab", "a", rows, {}) + assert total == 2 + expected_entries = [ + f"{g}/ ({c} {'doc' if c == 1 else 'docs'})" for g, c, _s in sub + ] + [path.rsplit("/", 1)[-1] for _src, path, _t, _d in files if path != suggested.path] + # Subfolder source-relative (``a/c/``), files by relative filename. + assert expected_entries == ["a/c/ (1 doc)", "a.md"] + expected_line = f"Homelab/a/: {', '.join(expected_entries)}" + assert agent.suggested_folder_lines(db, [suggested]) == [expected_line] + + +def test_suggested_folder_lines_empty_suggested_yields_no_lines( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """No suggested docs → no lines (the caller then builds the + byte-identical phase-118 prompt) — and no fetcher call at all.""" + calls: list[str] = [] + + def _rows(db: Any, source: str) -> list[tuple[str, str, str]]: + calls.append(source) + return [] + + monkeypatch.setattr(agent, "_source_document_rows", _rows) + monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {}) + assert agent.suggested_folder_lines(cast("Session", object()), []) == [] + assert calls == [] + + # ---------- the pinned drill-down templates (byte-for-byte) ---------- diff --git a/tests/unit/test_chat_cancel.py b/tests/unit/test_chat_cancel.py index db01d37..99ef0b7 100644 --- a/tests/unit/test_chat_cancel.py +++ b/tests/unit/test_chat_cancel.py @@ -183,6 +183,13 @@ class _FakeSession: return KbOverview(id=1, content="") return None + def execute(self, *args: Any, **kwargs: Any) -> list[Any]: + # Phase 119 (D3): the endpoint's suggested-folder lines run the + # ls catalog fetchers on this session — the fake catalog is + # empty (header-only lines), keeping the prompt builds + # deterministic here. + return [] + @pytest.fixture() def env(monkeypatch: pytest.MonkeyPatch) -> Iterator[_FakeSession]: @@ -411,10 +418,16 @@ def test_completed_turn_still_emits_done_and_writes_query_log( frames = _frames(chunks) assert [f["type"] for f in frames] == ["delta", "delta", "delta", "done"] assert frames[-1]["deflected"] is False - assert frames[-1]["sources"][0]["title"] == "Kubernetes Homelab Cluster" + # Phase 119 (LOCKED A1): the citation surface is the agent's READ + # docs only — this turn's model stream never emits a tool call, so + # nothing was read and the grounded done frame chips nothing (the + # phase-118 "suggested + read" union is retired). The retrieval + # stays durably recorded (118-A3 stands). + assert frames[-1]["sources"] == [] (row,) = env.added assert isinstance(row, QueryLog) assert row.question == "How is my Kubernetes cluster set up?" + assert "kubernetes-homelab-cluster.md" in row.sources # durable record kept assert env.commits == 1 # The per-turn line still goes out; no cancel line for a settled turn. assert any("question=" in r.getMessage() for r in caplog.records) diff --git a/tests/unit/test_chat_gate.py b/tests/unit/test_chat_gate.py index b394bb3..2c7efdd 100644 --- a/tests/unit/test_chat_gate.py +++ b/tests/unit/test_chat_gate.py @@ -11,7 +11,7 @@ from __future__ import annotations import hashlib import json import uuid -from collections.abc import Iterator +from collections.abc import Iterator, Sequence from datetime import UTC, datetime from typing import TYPE_CHECKING, Any @@ -22,9 +22,10 @@ from app.api import chat as chat_api from app.config import Settings from app.main import app as fastapi_app from app.models import Document, KbOverview, QueryLog +from app.rag import agent as rag_agent from app.rag.agent import AGENT_TOOLS -from app.rag.llm import StreamPiece -from app.rag.prompts import build_deflect_prompt +from app.rag.llm import StreamPiece, ToolCallPiece +from app.rag.prompts import SUGGEST_INTRO, build_deflect_prompt from app.rag.retriever import RetrievedChunk, weak_hit_titles from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions from tests.conftest import ADMIN_PASSWORD @@ -443,6 +444,51 @@ def test_plan_turn_high_seeds_top5_suggested_related_is_rank6plus() -> None: assert f"FULL_CONTENT_SENTINEL_{i}" not in plan.system_prompt +def test_plan_turn_folder_lines_ride_the_high_prompt() -> None: + """Phase 119 (D3, LOCKED A4): plan_turn passes *folder_lines* through + to the HIGH prompt — after ``SUGGEST_INTRO``, before the first + ```` block (each on its own line).""" + doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT") + lines = ["Homelab/: kubernetes-homelab-cluster.md (2 docs)"] + plan = chat_api.plan_turn( + [_chunk(doc, 0.90)], _settings(threshold=0.30), folder_lines=lines + ) + assert plan.deflected is False + prompt = plan.system_prompt + i_open = prompt.index("") + i_block = prompt.index("\n{SUGGEST_INTRO}\n{lines[0]}\n\n" + + +def test_plan_turn_folder_lines_default_keeps_phase_118_high_prompt() -> None: + """Omitted *folder_lines* (the default ``()``) ⇒ the HIGH prompt is + the phase-118 shape, byte-identical.""" + doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT") + plan = chat_api.plan_turn([_chunk(doc, 0.90)], _settings(threshold=0.30)) + assert plan.deflected is False + prompt = plan.system_prompt + i_open = prompt.index("") + i_block = prompt.index("\n{SUGGEST_INTRO}\n\n" + + +def test_plan_turn_folder_lines_ignored_on_the_low_branch() -> None: + """The LOW (deflected) branch IGNORES *folder_lines* — the + deflected prompt stays byte-identical (LOCKED A4).""" + doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT") + low_with = chat_api.plan_turn( + [_chunk(doc, 0.05, cosine=0.05)], + _settings(threshold=0.30), + folder_lines=["Homelab/: x.md"], + ) + low_without = chat_api.plan_turn( + [_chunk(doc, 0.05, cosine=0.05)], _settings(threshold=0.30) + ) + assert low_with.deflected is True + assert low_with.system_prompt == low_without.system_prompt + assert "Homelab/: x.md" not in low_with.system_prompt + + def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None: """The suggested cap is a CEILING, not a quota: one doc ⇒ one suggested doc, an empty related tier (nothing beyond rank 1).""" @@ -458,10 +504,11 @@ def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None: def test_plan_turn_high_suggests_strong_and_weak_no_floor() -> None: """The recurring incident under phase 118 (A3): the weak 2nd doc no - longer loses a citation slot to a bar — the floor never filters, so - BOTH docs are suggested (rank order) and ride the citation surface - (A4); the HIGH prompt seeds both summaries (the A5 fallback carries - the short fixture content whole).""" + longer loses a seeding slot to a bar — the floor never filters, so + BOTH docs are suggested (rank order) and ride the durable record + (118-A3 — the phase-119 A1 citation surface is the read docs only); + the HIGH prompt seeds both summaries (the A5 fallback carries the + short fixture content whole).""" strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT") weak = _doc("Backup Strategy", "WEAK_DOC_CONTENT") chunks = [_chunk(strong, 0.90, cosine=0.80), _chunk(weak, 0.80, cosine=0.20)] @@ -845,15 +892,21 @@ def test_suggestions_empty_input_yields_fallback_only() -> None: class _CannedLLM: """Records the messages it is given; streams a canned answer. - Never emits tool calls, so a grounded turn through the phase-37 agent - loop ends after the single (tools-offered) request; *seen_tools* - records each request's ``tools`` value for the phase-37 wiring pins. + Without *read_paths* it never emits tool calls, so a grounded turn + through the phase-37 agent loop ends after the single (tools-offered) + request. With *read_paths*, the request whose conversation carries + *i* tool results (i < len(read_paths)) emits ``read(read_paths[i])`` + (phase 119: the read-only done-sources pins drive multi-read turns), + and once the list is exhausted the request streams the answer. + *seen_tools* records each request's ``tools`` value for the phase-37 + wiring pins. """ - def __init__(self, answer: str = ANSWER) -> None: + def __init__(self, answer: str = ANSWER, read_paths: Sequence[str] = ()) -> None: self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue] self.embed_batches = 0 self.answer = answer + self.read_paths = list(read_paths) self.seen: list[list[dict[str, str]]] = [] self.seen_tools: list[list[dict[str, Any]] | None] = [] @@ -868,6 +921,15 @@ class _CannedLLM: ): self.seen.append(messages) self.seen_tools.append(tools) + if tools is not None and self.read_paths: + tool_results = sum(1 for m in messages if m.get("role") == "tool") + if tool_results < len(self.read_paths): + yield ToolCallPiece( + id=f"call_{tool_results + 1}", + name="read", + arguments={"path": self.read_paths[tool_results]}, + ) + return for i in range(0, len(self.answer), 12): yield StreamPiece("content", self.answer[i : i + 12]) @@ -913,6 +975,15 @@ class _FakeSession: return KbOverview(id=1, content=self.kb_overview) return None + def execute(self, *args: Any, **kwargs: Any) -> list[Any]: + """Phase 119 (D3): the endpoint's suggested-folder lines run the + ``ls`` catalog fetchers (``_source_document_rows`` / + ``_source_folder_summaries``) on this session — the fake catalog + is empty, so each suggested doc's line is its header alone + (``Homelab/:``), keeping the prompt builds deterministic in + these gate tests.""" + return [] + @pytest.fixture(autouse=True) def _admin_signed_in(client: TestClient) -> None: @@ -972,6 +1043,16 @@ def _fake_retriever(chunks: list[RetrievedChunk]) -> Any: return retrieve +def _find_docs(docs: dict[tuple[str, str], Document]) -> Any: + """A fake ``find_document`` resolving *docs* by ``(source, path)`` + (the agent module's hook — the fake sessions here have no catalog).""" + + def find_document(_db: Any, source: str, path: str) -> Document | None: + return docs.get((source, path)) + + return find_document + + def test_endpoint_just_below_threshold_deflects( client: TestClient, gate_env: tuple[_FakeSession, _CannedLLM], @@ -1073,11 +1154,18 @@ def test_endpoint_deflected_turn_never_offers_tools( assert "You may extend your context with three tools" not in system["content"] -def test_endpoint_score_at_threshold_answers( +def test_endpoint_score_at_threshold_answers_zero_read_grounded_turn_chips_nothing( client: TestClient, gate_env: tuple[_FakeSession, _CannedLLM], monkeypatch: pytest.MonkeyPatch, ) -> None: + """Phase 119 (LOCKED A1 — the phase-118 "always chips" pin + re-targeted): a grounded turn on which the agent reads NOTHING (the + summary-only fast path — the canned LLM never emits a tool call) + chips nothing: ``done.sources`` is the READ DOCS only, so the + never-read suggested doc appears NOWHERE in it (the explicit contrast + against the retired A4 union). The durable record keeps the retrieval + (118-A3 stands).""" session, llm = gate_env doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT") monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.30)])) @@ -1088,17 +1176,152 @@ def test_endpoint_score_at_threshold_answers( assert done["type"] == "done" assert done["deflected"] is False assert done["suggestions"] == [] - assert done["sources"] and done["sources"][0]["title"] == "Kubernetes Homelab Cluster" + # A1: read docs only — nothing was read ⇒ no chips. The suggested + # doc (rank 1) appears NOWHERE in done.sources (the retired A4 union + # would have carried it). + assert done["sources"] == [] (system, _user) = llm.seen[0][0], llm.seen[0][1] assert "HIGH" in system["content"] assert "DEFLECT_MODE" not in system["content"] assert "TALOS_DOC_SENT" in system["content"] + # The durable record still records the retrieval (118-A3 stands). (row,) = session.added assert isinstance(row, QueryLog) assert row.deflected is False - assert row.top_score == pytest.approx(0.30) + assert "kubernetes-homelab-cluster.md" in row.sources + + +def test_endpoint_done_sources_are_the_read_docs_in_read_order( + client: TestClient, + gate_env: tuple[_FakeSession, _CannedLLM], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Phase 119 (LOCKED A1): the done frame's ``sources`` is exactly + the agent's READ documents — deduped by (source, path), READ order + (not the suggested tier's rank order). Suggested docs that were + never read appear NOWHERE in the frame (the contrast pin against + the retired A4 union); the durable record keeps the full retrieval + (118-A3 stands).""" + session, _llm = gate_env + a = _doc("Alpha", "ALPHA_DOC_CONTENT") + b = _doc("Beta", "BETA_DOC_CONTENT") + c = _doc("Gamma", "GAMMA_DOC_CONTENT") + chunks = [_chunk(a, 0.90), _chunk(b, 0.80), _chunk(c, 0.70)] + monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks)) + llm = _CannedLLM(read_paths=[f"Homelab/{b.path}", f"Homelab/{a.path}"]) + fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: llm + monkeypatch.setattr( + rag_agent, + "find_document", + _find_docs({(a.source, a.path): a, (b.source, b.path): b}), + ) + + frames = _ask(client, "Which of these documents do I have?") + + done = frames[-1] + assert done["deflected"] is False + # Both reads streamed as tool frames (the agent loop ran). + reads = [f for f in frames if f["type"] == "tool"] + assert [f["argument"] for f in reads] == [ + f"Homelab/{b.path}", + f"Homelab/{a.path}", + ] + # A1: read docs only, read order — B first (it was read first), A + # second; the never-read suggested doc C is NOWHERE in the frame, and + # the suggested tier's rank order (A, B, C) is NOT the chip order. + assert [(s["source"], s["path"]) for s in done["sources"]] == [ + (b.source, b.path), + (a.source, a.path), + ] + assert not any(s["path"] == c.path for s in done["sources"]) + assert len(done["sources"]) == len({(s["source"], s["path"]) for s in done["sources"]}) + assert done["related"] == [] # nothing beyond rank 3 for 3 docs + + # The durable record keeps the full retrieval (118-A3 stands). + (row,) = session.added + assert isinstance(row, QueryLog) + for doc in (a, b, c): + assert f"Homelab/{doc.path}" in row.sources + + +def test_endpoint_read_related_doc_is_cited_not_related( + client: TestClient, + gate_env: tuple[_FakeSession, _CannedLLM], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Phase 119 (A1) × phase 118 (A3): a related-tier doc (rank 6+) that + the agent ``read``s is a CHIP (in ``done.sources``) and is EXCLUDED + from ``done.related`` — a "nearby doc" that was actually used must + not read as nearby (unchanged intent, keyed on read docs since + phase 119). The suggested docs, never read, are absent from the + frame; the other related doc stays in the tier.""" + session, _llm = gate_env + docs = [_doc(f"Doc {i}", f"CONTENT_{i}") for i in range(7)] + chunks = [ + _chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs) + ] + monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks)) + read_doc = docs[5] # rank 6 — the related tier (rank 7 = docs[6]) + llm = _CannedLLM(read_paths=[f"Homelab/{read_doc.path}"]) + fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: llm + monkeypatch.setattr( + rag_agent, + "find_document", + _find_docs({(read_doc.source, read_doc.path): read_doc}), + ) + + frames = _ask(client, "What does the sixth ranked document say?") + + done = frames[-1] + assert done["deflected"] is False + # A1: read docs only — the read related doc is the ONLY chip (the + # never-read suggested docs are nowhere in the frame). + assert [(s["source"], s["path"]) for s in done["sources"]] == [ + (read_doc.source, read_doc.path) + ] + # The read related doc is NOT "nearby"; the other related doc is. + related = [(s["source"], s["path"]) for s in done["related"]] + assert related == [(docs[6].source, docs[6].path)] + assert not any((d.source, d.path) in related for d in docs[:5]) + + # The durable record keeps the full retrieval (118-A3 stands). + (row,) = session.added + assert isinstance(row, QueryLog) + assert row.deflected is False + for d in docs: + assert f"Homelab/{d.path}" in row.sources + + +def test_endpoint_suggested_folder_lines_reach_the_high_prompt( + client: TestClient, + gate_env: tuple[_FakeSession, _CannedLLM], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Phase 119 (D3, LOCKED A4): the endpoint computes the + suggested-folder lines BEFORE plan_turn (the same deterministic + suggested walk, over the short-lived step session) and they ride + the grounded HIGH prompt — after ``SUGGEST_INTRO``, before the + first ```` block.""" + _session, llm = gate_env + doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT") + monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.30)])) + line = "Homelab/: kubernetes-homelab-cluster.md (2 docs)" + monkeypatch.setattr( + chat_api, + "suggested_folder_lines", + lambda db, suggested: [line], + ) + frames = _ask(client, "How is my Kubernetes cluster set up?") + assert frames[-1]["deflected"] is False + (system, _user) = llm.seen[0][0], llm.seen[0][1] + i_open = system["content"].index("") + i_block = system["content"].index("\n{SUGGEST_INTRO}\n{line}\n\n" + ) # ---------- endpoint: KB overview row (phase 31) ---------- diff --git a/tests/unit/test_config.py b/tests/unit/test_config.py index 2cf9978..ee4f73c 100644 --- a/tests/unit/test_config.py +++ b/tests/unit/test_config.py @@ -41,6 +41,10 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non # fine-line-tuned value, task 07) with a 365-day decay timescale. assert s.recency_boost == 0.0007 assert s.recency_half_life_days == 365 + # Phase 119, D2 (LOCKED A3): the bounded name-hit bonus on the + # selection-time document score is ON by default (0.005 — the + # owner-tunable starting point; ``0`` is the kill switch). + assert s.name_hit_bonus == 0.005 # Owner instruction 2026-08-22: answers may run up to 32 768 tokens. assert s.max_output_tokens == 32_768 # Phase 17: the model's thinking streams by default (kill-switch off). @@ -391,6 +395,33 @@ def test_recency_boost_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None _settings() +def test_name_hit_bonus_default_and_env_override( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Phase 119, D2 (LOCKED A3): the bounded name-hit bonus on the + SELECTION-time document score is ON by default (0.005 — the + owner-tunable starting point, not a calibrated constant) and + env-tunable so the owner re-tunes live (the phase-106 recency-boost + precedent). ``0`` is legal — the byte-identical kill switch.""" + monkeypatch.delenv("BOR_NAME_HIT_BONUS", raising=False) + s = _settings() + assert s.name_hit_bonus == 0.005 + monkeypatch.setenv("BOR_NAME_HIT_BONUS", "0") + assert _settings().name_hit_bonus == 0.0 + monkeypatch.setenv("BOR_NAME_HIT_BONUS", "0.01") + assert _settings().name_hit_bonus == 0.01 + + +def test_name_hit_bonus_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None: + """``0`` is the byte-identical kill switch (the pre-phase selection + order) — a NEGATIVE bonus would demote name-hit documents (the exact + opposite of D2), so the validator fails loudly at startup naming the + field (the ``agent_max_rounds`` pattern, phase 119).""" + monkeypatch.setenv("BOR_NAME_HIT_BONUS", "-0.001") + with pytest.raises(ValidationError, match="name_hit_bonus"): + _settings() + + def test_recency_half_life_rejects_non_positive( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/tests/unit/test_embed_question_length.py b/tests/unit/test_embed_question_length.py index c1e5249..7f8875f 100644 --- a/tests/unit/test_embed_question_length.py +++ b/tests/unit/test_embed_question_length.py @@ -173,6 +173,13 @@ class _FakeSession: return None return None + def execute(self, *args: Any, **kwargs: Any) -> list[Any]: + # Phase 119 (D3): the endpoint's suggested-folder lines run the + # ls catalog fetchers on this session — the fake catalog is + # empty (header-only lines), keeping the prompt builds + # deterministic here. + return [] + def _doc(title: str, content: str) -> Document: return Document( diff --git a/tests/unit/test_prompt_lock.py b/tests/unit/test_prompt_lock.py index eb59cd9..45f3937 100644 --- a/tests/unit/test_prompt_lock.py +++ b/tests/unit/test_prompt_lock.py @@ -148,16 +148,27 @@ def test_low_prompt_build_byte_identical_to_pre_phase() -> None: assert "SUGGEST_INTRO" not in prompt and "" not in prompt -# ---------- SUGGEST_INTRO (phase 118, task 03 — the start-here framing) ---------- +# ---------- SUGGEST_INTRO (phase 118 framing, re-cut for phase 119, LOCKED A5) ---------- -#: Phase-118 anchors for ``SUGGEST_INTRO`` — the ```` -#: section's intro line (the owner's "start here if these summaries seem -#: right to you" framing, TODO L3). The E2E mock's ``_document_block`` -#: parser is regex-based over the block markup (which stays byte-stable -#: around the intro), so this constant is a prompt-copy lock, pinned the -#: way ``TOOLS_SECTION`` is: sha256 + prefix + total length. -SUGGEST_INTRO_SHA256 = "7b14d2dedc6ebc4e440d32dd1edb979a7461c94034b9541c9f37b9042f394d3a" -SUGGEST_INTRO_LEN = 323 +#: Anchors for ``SUGGEST_INTRO`` — the ```` section's intro +#: line (the owner's "start here if these summaries seem right to you" +#: framing, TODO L3, phase 118). RE-CUT for phase 119 (task 04, LOCKED +#: A5, owner directive 2026-09-16): the final sentence was replaced +#: with the cite-discipline sentence (cite only the documents read — or +#: the suggested document answered from without reading — never a +#: document neither read nor used), closing the live confabulation in +#: which the model cited a file it never read; the phase-118 "cite the +#: document(s) you used, by path" sentence is retired. Only that +#: sentence moved — the start-here framing, the ``read`` pointer, and +#: the full-text clause survive byte-identical, so the prefix anchor is +#: the same as pre-phase-119. The E2E mock's ``_document_block`` parser +#: is regex-based over the block markup (which stays byte-stable around +#: the intro), so this constant is a prompt-copy lock, pinned the way +#: ``TOOLS_SECTION`` is: sha256 + prefix + total length. ``PERSONA``, +#: ``TOOLS_SECTION``, and the LOW body anchors above stay byte-identical +#: — this change touches no other constant. +SUGGEST_INTRO_SHA256 = "c6ffbb19975cbf184910a9a8e8fdea101c16f5d3eecb872fb07afeabd2763302" +SUGGEST_INTRO_LEN = 460 SUGGEST_INTRO_PREFIX = ( "The blocks below are the summaries of the top-ranked documents for " "your question — start here if one seems right to you: " @@ -165,14 +176,23 @@ SUGGEST_INTRO_PREFIX = ( def test_suggest_intro_byte_locked() -> None: - """The start-here framing is LOCKED copy (phase 118): sha256 + exact - prefix + total length; the three contracts it must carry (summaries - are the starting points; ``read`` adds the full text, which is NOT - in the prompt until read; cite by path) are pinned as substrings.""" + """The start-here framing is LOCKED copy (phase 118; the final + sentence re-cut by phase 119, LOCKED A5): sha256 + exact prefix + + total length; the four contracts it must carry (summaries are the + starting points; ``read`` adds the full text, which is NOT in the + prompt until read; the phase-119 cite discipline — cite only what + was read, or the suggested document answered from without reading, + never a document neither read nor used) are pinned as substrings.""" assert len(SUGGEST_INTRO) == SUGGEST_INTRO_LEN assert _sha256(SUGGEST_INTRO) == SUGGEST_INTRO_SHA256 assert SUGGEST_INTRO.startswith(SUGGEST_INTRO_PREFIX) - assert "call `read`" in SUGGEST_INTRO + assert "call `read" in SUGGEST_INTRO assert "combined `source/path`" in SUGGEST_INTRO assert "its full text is not in the prompt until you read it" in SUGGEST_INTRO - assert "Cite the document(s) you used, by path." in SUGGEST_INTRO + assert ( + "Cite only the document(s) you read — or, if you answered from a " + "suggested summary without reading it, cite that suggested " + "document — never a document you neither read nor used." + ) in SUGGEST_INTRO + # The retired phase-118 final sentence is gone from the constant. + assert "Cite the document(s) you used, by path." not in SUGGEST_INTRO diff --git a/tests/unit/test_prompts.py b/tests/unit/test_prompts.py index a460543..47776cf 100644 --- a/tests/unit/test_prompts.py +++ b/tests/unit/test_prompts.py @@ -24,6 +24,13 @@ document is refused — while the ``ls``/``grep`` clauses and the discipline rules stay byte-identical): the teaching refusals in :mod:`app.rag.agent` re-state the same contract; the ```` marker keying (HIGH only) is unchanged. + +And the phase-119 cite discipline (task 04, LOCKED A5): the intro's +final sentence ("cite the document(s) you used, by path", phase 118) +is REPLACED — the HIGH prompt carries the discipline sentence exactly +once, inside ```` after the intro's start-here framing; +the LOW prompt and ``PERSONA`` / ``TOOLS_SECTION`` stay byte-identical +(the full sha pins live in :mod:`tests.unit.test_prompt_lock`). """ from __future__ import annotations @@ -447,6 +454,155 @@ def test_documents_section_without_blocks_has_no_intro() -> None: ) +# ---------- phase 119 (D3, LOCKED A4): the suggested-folder lines ---------- + + +def test_high_prompt_folder_lines_after_intro_before_first_block() -> None: + """The folder lines ride the ```` section immediately + AFTER the ``SUGGEST_INTRO`` line — each on its own line — then a + blank line, then the first ```` block. Plain lines: no + new markup/tag anywhere (the E2E mock keys off the ```` + marker and the LAST block's tail).""" + doc = _doc( + "deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2", + "FULL_CONTENT_SENTINEL_119", + "Gitea Web Env", + summary="Gitea web env file.", + ) + lines = [ + "Homelab/deploy/Deployments/reeseapps/gitea/: app/ (5 docs), README.md", + "Homelab/deploy/Deployments/reeseapps/: gateway/ (3 docs), gitea/ (8 docs)", + ] + prompt = build_high_prompt([doc], folder_lines=lines) + i_open = prompt.index("") + i_block = prompt.index("\n{SUGGEST_INTRO}\n{lines[0]}\n{lines[1]}\n\n" + # Each line occurs exactly once, inside the section, before the + # first block (never after a summary — the E2E tail echo is safe). + for line in lines: + assert prompt.count(line) == 1 + assert prompt.index("") < prompt.index(line) < prompt.index("") + # The block markup AND body stay byte-identical: from the + # first block on, the prompt equals the no-folder-line build. + plain = build_high_prompt([doc]) + assert prompt[prompt.index("`` MENTION in the TOOLS_SECTION copy is plain text — + # count the tag + newline, not the bare substring). + assert prompt.count("\n") == 1 + assert prompt.count("") == 1 + assert prompt.count(" None: + """One folder line: the exact slice between the section open and the + first block is ``\n`` + intro + ``\n`` + line + + ``\n\n``.""" + doc = _doc("a.md", "CONTENT", "Title A", summary="Summary A.") + line = "Homelab/: a.md (2 docs), b/ (1 doc)" + prompt = build_high_prompt([doc], folder_lines=[line]) + i_open = prompt.index("") + i_block = prompt.index("\n{SUGGEST_INTRO}\n{line}\n\n" + + +def test_high_prompt_empty_folder_lines_byte_identical_to_phase_118() -> None: + """LOCKED A4: empty ``folder_lines`` (the default) ⇒ the phase-118 + build, byte-identical — the existing pins keep passing and this + pins the default explicitly (both an explicit ``()`` and the + omitted parameter).""" + doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster") + block = ( + '\n' + "Talos Linux on three nodes.\n" + "" + ) + expected = ( + _base("HIGH") + + "\n\n" + + SUGGEST_INTRO + + "\n\n" + + block + + "\n" + + "\n" + + TOOLS_SECTION + ) + # An explicit empty tuple AND the omitted parameter (the default): + assert build_high_prompt([doc], folder_lines=()) == expected + assert build_high_prompt([doc]) == expected + # The LOW (deflected) prompt has no folder_lines parameter at all — + # the deflected build stays byte-identical (the phase-118 pin in + # test_zero_note_prompt_is_byte_identical_to_pre_steering stands). + + +def test_folder_lines_ride_only_on_present_blocks() -> None: + """Like the intro, the folder lines ride on present blocks ONLY: an + empty ```` section is unchanged (no lines, no blocks).""" + prompt = build_high_prompt([], folder_lines=["X/: y.md"]) + assert prompt == build_high_prompt([]) + assert "X/: y.md" not in prompt + + +# ---------- phase 119 (D4, LOCKED A5): the cite-discipline sentence ---------- + +#: The LOCKED A5 sentence (phase 119, task 04) — the exact replacement +#: for the retired phase-118 final sentence of :data:`SUGGEST_INTRO` +#: ("Cite the document(s) you used, by path."). Closes the live +#: confabulation: the answer's "Docs used:" line cited a file the +#: agent never read. +CITE_DISCIPLINE = ( + "Cite only the document(s) you read — or, if you answered from a " + "suggested summary without reading it, cite that suggested " + "document — never a document you neither read nor used." +) + + +def test_high_prompt_carries_cite_discipline_exactly_once_in_documents() -> None: + """LOCKED A5: the HIGH prompt carries the discipline sentence + EXACTLY ONCE — it IS the intro's final sentence, so it sits inside + ````, AFTER the intro's start-here framing and BEFORE + the first block (both the plain build and the folder-line build; + the sentence rides the intro line, which never moves).""" + doc = _doc( + "kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster", summary="K8S" + ) + assert CITE_DISCIPLINE in SUGGEST_INTRO + assert SUGGEST_INTRO.endswith(CITE_DISCIPLINE) # it is the final sentence + for high in (build_high_prompt([doc]), build_high_prompt([doc], folder_lines=["X/: y.md"])): + assert high.count(CITE_DISCIPLINE) == 1 + i_docs = high.index("") + i_sentence = high.index(CITE_DISCIPLINE) + i_block = high.index("") + assert i_docs < i_sentence < i_close + assert i_sentence > high.index("start here if one seems right") + assert i_sentence < i_block # before the first block, never after a summary + # The retired phase-118 sentence is gone from the prompt (it is + # gone from the constant — pinned in test_prompt_lock too). + high = build_high_prompt([doc]) + assert "Cite the document(s) you used, by path." not in high + # The sentence rides on present blocks ONLY: an empty ```` + # section (no intro) carries none of it. + assert CITE_DISCIPLINE not in build_high_prompt([]) + + +def test_cite_discipline_absent_from_low_prompt() -> None: + """LOCKED A5: the discipline sentence belongs to ``SUGGEST_INTRO`` + (the HIGH path) — it never leaks into the LOW (deflected) prompt, + whose byte-identity is pinned separately (the LOW anchors in + ``test_prompt_lock`` pass unchanged).""" + for prompt in ( + build_deflect_prompt(["T1", "T2"]), + build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW), + build_deflect_prompt([]), + ): + assert CITE_DISCIPLINE not in prompt + assert "Cite only the document(s) you read" not in prompt + assert "Cite the document(s) you used, by path." not in prompt + + def test_tools_section_old_names_and_budget_copy_gone() -> None: """The phase-37/68 tool names and the phase-37 per-tool budget line (phase 45: the round cap is the bound — the prompt does not diff --git a/tests/unit/test_retriever.py b/tests/unit/test_retriever.py index 8514174..cf6357a 100644 --- a/tests/unit/test_retriever.py +++ b/tests/unit/test_retriever.py @@ -721,14 +721,15 @@ def _lexical_row(is_summary: bool, doc_path: str) -> object: def test_lexical_candidates_carry_is_summary_flag() -> None: """The lexical list reads ``c.is_summary`` from the raw row. - The question carries no digit-bearing name token (no bare, no - numeric-join), so the name-hit path issues NO queries at all — the - single FTS rowset answers the only (FTS) call, and the list is the - plain FTS rows: the pre-name-hit behavior, unchanged. + The question's tokens are name candidates (class-agnostic, phase + 119), so the name-hit projection runs — but the (empty) catalog + yields no path match, the LATERAL fetch is skipped, and the FTS + rowset answers the second call: the list is the plain FTS rows, + every one ``name_hit=False``. """ rows = [_lexical_row(True, "summary-src.yaml"), _lexical_row(False, "other.md")] out = _lexical_candidates( - _FakeSession(rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType] + _FakeSession([], rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType] ) assert len(out) == 2 by_path = {rc.document.path: rc for rc in out} @@ -736,6 +737,7 @@ def test_lexical_candidates_carry_is_summary_flag() -> None: assert by_path["summary-src.yaml"].position == -1 assert by_path["other.md"].is_summary is False assert all(rc.fts_hit is True for rc in out) + assert all(rc.name_hit is False for rc in out) # ordinary FTS rows def test_fuse_keeps_is_summary_on_double_hit() -> None: @@ -783,30 +785,54 @@ def test_normalize_name() -> None: def test_name_hit_tokens_incident_question() -> None: - """The incident question yields EXACTLY the versioned join - ``qwen38`` — the token the document names actually carry. Plain - prose words (``what``, ``llamacpp``, ``arguments``, ``server`` — - no digit) never name-match (the precision guard); the single - digits ("3", "8") and the bare "38" are < 4 chars; the - digit-leading ``38show`` boundary artifact is dropped.""" + """Phase 119 (LOCKED A2): the candidate list is CLASS-AGNOSTIC — + every normalized token of length >= 4 (dotted kept whole: + ``llama.cpp`` → ``llamacpp``) plus the versioned join ``qwen38``. + The digit distinction moved to the match side (:func:`_name_hit_chunks`) — + prose precision now comes from the match class (a digitless token + must EQUAL a whole path component). The single digits ("3", "8") + and the bare "38" are < 4 chars; the digit-leading ``38show`` + boundary artifact cannot survive (the join only fires on a purely + numeric SECOND token).""" tokens = name_hit_tokens(INCIDENT_QUESTION) - assert tokens == ["qwen38"] - for absent in ("what", "qwen", "llamacpp", "arguments", "3", "8", "38", "38show", "server"): + assert tokens == ["what", "correct", "llamacpp", "arguments", "qwen", "qwen38"] + for absent in ("3", "8", "38", "38show", "server"): assert absent not in tokens -def test_name_hit_tokens_no_digit_question_returns_empty() -> None: - """A question with no digit-bearing token (bare or joined) yields - no name candidates — prose joins like ``correctllama`` never count.""" - assert name_hit_tokens("what is the correct caddy config") == [] +def test_name_hit_tokens_digitless_question_yields_long_tokens() -> None: + """A question with NO digit-bearing token still yields candidates + (every normalized token of length >= 4) — the 2026-09-16 fix: + product names without digits ("gitea", "gateway") must get a name + signal. Prose joins (``correctcaddy``) never count (the second + token is not purely numeric).""" + assert name_hit_tokens("what is the correct caddy config") == [ + "what", "correct", "caddy", "config", + ] assert name_hit_tokens("a e i o u 3 8") == [] def test_name_hit_tokens_bare_digit_bearing_token() -> None: """A single written token that carries a digit (``1panel``) is a - name candidate on its own — no join needed.""" + name candidate on its own — no join needed — alongside the plain + prose tokens of the same question (class-agnostic list).""" tokens = name_hit_tokens("what is my 1panel dashboard setup") - assert tokens == ["1panel"] + assert tokens == ["what", "1panel", "dashboard", "setup"] + + +def test_name_hit_tokens_versioned_join_and_short_tokens() -> None: + """The versioned join survives the class-agnostic change ("Qwen + 3.8" → ``qwen38``), and short tokens (< :data:`NAME_TOKEN_MIN_LEN` + normalized — the single digits, "3.8" → ``38``) never become + candidates, with or without a join.""" + tokens = name_hit_tokens("help me with Qwen 3.8 please") + # The join is appended at its FIRST token's position (after "qwen"). + assert tokens == ["help", "with", "qwen", "qwen38", "please"] + assert name_hit_tokens("3.8 8 16 9") == [] # 38 / 8 / 16 / 9 / 816 / 169 all < 4 + # Word-after-version: the word itself is a candidate, but the + # digit-leading join artifact ("38show") cannot survive (the join + # only fires on a purely numeric SECOND token). + assert name_hit_tokens("3.8 show") == ["show"] def _name_row(doc: Document) -> tuple: @@ -834,7 +860,8 @@ def _name_hit_lateral_row(doc: Document, is_summary: bool = False) -> SimpleName def test_name_hit_chunks_no_tokens_skips_all_queries() -> None: - """A question with no name tokens issues no queries at all.""" + """A question with no name tokens (every normalized token < 4) + issues no queries at all.""" session = _FakeSession([]) # any call would surface a statement assert _name_hit_chunks(session, "a e i o u 3 8") == [] # pyright: ignore[reportArgumentType] assert session.statements == [] @@ -850,46 +877,124 @@ def test_name_hit_chunks_no_matching_doc_returns_empty() -> None: assert len(session.statements) == 1 # projection only — no LATERAL fetch -def test_name_hit_chunks_ranked_by_count_length_catalog() -> None: - """A two-candidate question (``qwen38`` + ``1panel``): the document - whose name carries BOTH (2 matches, 12 total chars) leads; the two - single-match documents tie on (1, 6) and fall to catalog order - (``dashboards/1panel-notes.md`` before ``quadlets/qwen3.8…``). - Hits carry ``fts_hit=True`` (the A8 gate answers), ``cosine=0.0``, - and the summary flag of their representative chunk.""" - both = _doc("dashboards/1panel-qwen3.8.md", "body", title="1Panel Qwen 3.8") - panel = _doc("dashboards/1panel-notes.md", "body") - q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body") - name_rows = [_name_row(d) for d in (panel, both, q38)] # catalog order - question = "what are the correct llama.cpp arguments for qwen 3.8 and the 1panel dashboard?" +def test_name_hit_chunks_digitless_exact_part_stem_subcomponent() -> None: + """A DIGITLESS token EQUALS a normalized path part (the ``gitea/`` + folder), the file stem (``gitea.md``), or a stem sub-component + (``kubernetes_gitea``, ``gitea-values``, ``test-gateway`` — the + stem split on non-alphanumeric runs) — the 2026-09-16 product-name + signal (LOCKED A2).""" + question = "how do i set up gitea or the gateway" # tokens: [gitea, gateway] + docs = [ + _doc("deploy/reeseapps/gitea/README.md", "body"), # path part + _doc("notes/gitea.md", "body"), # file stem + _doc("deploy/k8s/kubernetes_gitea.md", "body"), # sub-component + _doc("deploy/k8s/gitea-values.yaml", "body"), # sub-component + _doc("deploy/istio/test-gateway.yaml", "body"), # sub-component (gateway) + _doc("notes/gitlab.md", "body"), # NO component matches — excluded + ] + name_rows = [_name_row(d) for d in docs] + session = _FakeSession(name_rows, [_name_hit_lateral_row(d) for d in docs[:5]]) + out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType] + # Five one-token hits, catalog order (source, path): + assert [rc.document.path for rc in out] == [ + "deploy/istio/test-gateway.yaml", + "deploy/k8s/gitea-values.yaml", + "deploy/k8s/kubernetes_gitea.md", + "deploy/reeseapps/gitea/README.md", + "notes/gitea.md", + ] + assert all(rc.name_hit is True for rc in out) + assert all(rc.fts_hit is True for rc in out) # the lexical signal + assert all(rc.cosine == 0.0 for rc in out) # no vector rank + + +def test_name_hit_chunks_digitless_title_never_matched() -> None: + """The owner-verified failure mode of the naive relaxation: a doc + under a ``Deployments/`` folder titled "Deployments" does NOT hit + the common token ``deploy`` (the part normalizes to + ``deployments`` ≠ ``deploy``), and a doc titled "Gitea" with no + gitea path component does NOT hit ``gitea`` — TITLES ARE NEVER + MATCHED (LOCKED A2).""" + question = "how do i deploy gitea" # tokens: [deploy, gitea] + docs = [ + _doc("Deployments/reeseapps/README.md", "body", title="Deployments"), + _doc("notes/internal-notes.md", "body", title="Gitea"), # title only + ] + session = _FakeSession([_name_row(d) for d in docs], []) + assert _name_hit_chunks(session, question) == [] # pyright: ignore[reportArgumentType] + assert len(session.statements) == 1 # projection only — no LATERAL fetch + + +def test_name_hit_chunks_digit_bearing_prefix_not_midword() -> None: + """A DIGIT-BEARING token is a PREFIX of a normalized part or stem + (``qwen38`` → ``qwen3.8-27b-epic-vulkan.container``) — a stem that + merely CONTAINS the token mid-word (``xqwen38y…``) does NOT hit; + sub-components are in the exact-match class only (LOCKED A2).""" + question = "what are the arguments for qwen 3.8" # tokens: what, arguments, qwen, qwen38 + hit = _doc("quadlets/qwen3.8-27b-epic-vulkan.container", "body") + miss = _doc("quadlets/xqwen38y-test.container", "body") # mid-word containment + name_rows = [_name_row(hit), _name_row(miss)] + session = _FakeSession(name_rows, [_name_hit_lateral_row(hit)]) + out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType] + assert [rc.document.path for rc in out] == [hit.path] + assert out[0].name_hit is True + + +def test_name_hit_chunks_ranked_by_count_then_catalog() -> None: + """A question (``deploy`` + ``gitea`` + ``qwen`` + ``qwen38``): + the document whose path carries BOTH a digitless component and a + digit-bearing prefix (2 matched tokens) leads; the two + single-token documents tie on count and fall to CATALOG ORDER — + the old total-matched-length tie-break is RETIRED (it would have + put the 6-char ``qwen38`` hit, ``quadlets/…``, before the 5-char + ``gitea`` hit, ``gitea/notes.md`` — the flip is pinned). The + "Deployments"-titled doc and the title-only "Gitea" doc never + appear (titles are never matched).""" + precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments") + gitea_notes = _doc("gitea/notes.md", "body", title="Internal notes") + both = _doc("gitea/qwen3.8-model.container", "body", title="The model quadlet") + q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body", title="juggernaut") + title_only = _doc("notes/internal-notes.md", "body", title="Gitea") + name_rows = [_name_row(d) for d in (precision, gitea_notes, both, q38, title_only)] + question = "how do i deploy gitea with qwen 3.8" lateral_rows = [ _name_hit_lateral_row(q38, is_summary=True), # LATERAL may return any order _name_hit_lateral_row(both), - _name_hit_lateral_row(panel), + _name_hit_lateral_row(gitea_notes), ] session = _FakeSession(name_rows, lateral_rows) out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType] assert [rc.document.path for rc in out] == [ - "dashboards/1panel-qwen3.8.md", # 2 matched tokens — leads - "dashboards/1panel-notes.md", # (1, 6) — catalog order - "quadlets/qwen3.8-27b-juggernaut-vulkan.container", # (1, 6) — after + "gitea/qwen3.8-model.container", # 2 matched tokens (gitea + qwen38) — leads + "gitea/notes.md", # 1 token (gitea, 5 chars) — catalog order beats quadlets + "quadlets/qwen3.8-27b-juggernaut-vulkan.container", # 1 token (qwen38, 6 chars) ] + assert all(rc.name_hit is True for rc in out) assert all(rc.fts_hit is True for rc in out) # the lexical signal assert all(rc.cosine == 0.0 for rc in out) # no vector rank assert all(rc.score == 0.0 for rc in out) # fuse fills the score by_path = {rc.document.path: rc for rc in out} + # The representative chunk keeps its summary flag (the LATERAL + # choice: is_summary DESC, position ASC — chunk 0 otherwise). assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].is_summary is True assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].position == -1 - assert by_path["dashboards/1panel-notes.md"].is_summary is False + assert by_path["gitea/notes.md"].is_summary is False + + +def test_name_hit_chunks_short_tokens_never_hit() -> None: + """Short tokens (< 4 normalized — "3.8" → ``38``, the single + digits) are never candidates, so they can never hit, with or + without the versioned join.""" + session = _FakeSession([]) # any call would surface a statement + assert _name_hit_chunks(session, "3.8 8 16 9") == [] # pyright: ignore[reportArgumentType] + assert session.statements == [] def test_name_hit_chunks_capped_at_limit() -> None: - """Twelve tied name hits (one matched token each) yield exactly - ``NAME_HIT_LIMIT`` of them — catalog order (the deterministic - tie-break).""" - docs = [_doc(f"quadlets/m{i:02d}.container", "body") for i in range(12)] - for d in docs: # give every document a name that carries the token - d.title = "qwen38 model i" + """Twelve tied name hits (one matched token each — the ``qwen38`` + stem prefix) yield exactly ``NAME_HIT_LIMIT`` of them — catalog + order (the deterministic tie-break).""" + docs = [_doc(f"quadlets/qwen3.8-m{i:02d}.container", "body") for i in range(12)] name_rows = [_name_row(d) for d in docs] # Only the ten winners (catalog order — the deterministic tie-break # of the twelve identical scores) reach the LATERAL fetch; the fake @@ -900,7 +1005,10 @@ def test_name_hit_chunks_capped_at_limit() -> None: session, "tell me about the qwen 3.8 models" # pyright: ignore[reportArgumentType] ) assert len(out) == NAME_HIT_LIMIT - assert [rc.document.path for rc in out] == [f"quadlets/m{i:02d}.container" for i in range(10)] + assert [rc.document.path for rc in out] == [ + f"quadlets/qwen3.8-m{i:02d}.container" for i in range(10) + ] + assert all(rc.name_hit is True for rc in out) def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None: @@ -933,7 +1041,7 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None: session = _FakeSession(name_rows, lateral_rows, fts_rows) out = _lexical_candidates(session, INCIDENT_QUESTION, limit=10) # pyright: ignore[reportArgumentType] assert len(out) == 3 # q38 (once), other (name hit), other (FTS chunk) - # Both name hits tie on (1, 6) — catalog order: "qwen3." (ASCII 46) + # Both name hits tie on count (1) — catalog order: "qwen3." (ASCII 46) # sorts before "qwen38" (ASCII 56). assert out[0].document.path == "quadlets/qwen3.8-27b-juggernaut-vulkan.container" assert out[1].document.path == "quadlets/qwen38-other.container" @@ -942,3 +1050,299 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None: rc.chunk_id for rc in out } == {q38_chunk, fts_rows[1].chunk_id, lateral_rows[1].chunk_id} assert all(rc.fts_hit is True for rc in out) + # Phase 119: the name-hit representative rows are flagged, the plain + # FTS row is not (the selection tier's bonus input, task 02). + assert out[0].name_hit is True + assert out[1].name_hit is True + assert out[2].name_hit is False + + +# --------------------------------------------------------------------------- +# Phase 119, D1 — the name_hit flag through fusion +# --------------------------------------------------------------------------- + + +def test_fuse_keeps_name_hit_on_lexical_only_hit() -> None: + """A name-hit row with no vector rank keeps ``name_hit=True`` + through the fusion (the ``replace()`` copy carries the field).""" + nh = _rc("gitea/README.md") + nh.name_hit = True + out = fuse([], [nh], k=60) + assert len(out) == 1 + assert out[0].name_hit is True + assert out[0].fts_hit is True + assert out[0].cosine == 0.0 + + +def test_fuse_or_s_name_hit_on_double_hit() -> None: + """A vector row that is ALSO the name hit's representative chunk + (the RRF merge dedupes by chunk id) keeps ``name_hit=True`` — the + merge ORs the flag in, so the selection tier (task 02) still sees + the name hit on the fused list.""" + v = _rc("gitea/README.md", cosine=0.9) + l1 = _rc("gitea/README.md", cosine=0.1) # the lexical copy of the same chunk + l1.chunk_id = v.chunk_id + l1.name_hit = True + out = fuse([v], [l1], k=60) + assert len(out) == 1 + assert out[0].name_hit is True + assert out[0].fts_hit is True + assert out[0].score == pytest.approx(2 / 61) + + +def test_fuse_default_name_hit_stays_false_for_ordinary_rows() -> None: + """Neither list flagged ⇒ fusion never invents a name-hit flag — + ordinary vector and FTS rows are ``name_hit=False``.""" + out = fuse([_rc("a.md", cosine=0.8)], [_rc("b.md", fts_hit=True)], k=60) + assert len(out) == 2 + assert all(rc.name_hit is False for rc in out) + + +# --------------------------------------------------------------------------- +# Phase 119, D2 — the bounded name-hit bonus (LOCKED A3) +# --------------------------------------------------------------------------- + +from app.rag.retriever import _selection_order, weak_hit_titles # noqa: E402 + + +def _bonus_rc( + doc: Document, + score: float, + cosine: float, + position: int = 0, + name_hit: bool = False, +) -> RetrievedChunk: + """One fused-list candidate (name-hit rows follow the D1 lexical + convention: ``cosine=0.0``, ``fts_hit=True``).""" + return RetrievedChunk( + chunk_id=uuid.uuid4(), + position=position, + content=doc.content[:20], + score=score, + document=doc, + cosine=cosine, + fts_hit=cosine == 0.0, + name_hit=name_hit, + ) + + +def _bonus_chunks() -> list[RetrievedChunk]: + """A mixed FUSED list — already in the ``fuse()`` key order + (−score, −cosine, path, position) — with one name-hit document + (``gitea/README.md``, the D1 convention: cosine 0.0) and ordinary + vector/FTS documents: ``a.md`` carries two chunks, and ``b.md`` / + ``c.md`` tie on the fused score (separated only by cosine). + + The pre-phase (bonus-0) document order this list walks — the golden + the kill switch must reproduce — is a (0.0200) → gitea (0.0160) → + b (0.0150, cos 0.7) → c (0.0150, cos 0.6) → d (0.0100). + """ + gitea = _doc("gitea/README.md", "G" * 50) + a = _doc("a.md", "A" * 50) + b = _doc("b.md", "B" * 50) + c = _doc("c.md", "C" * 50) + d = _doc("d.md", "D" * 50) + return [ + _bonus_rc(a, 0.0200, 0.9, 0), + _bonus_rc(gitea, 0.0160, 0.0, 0, name_hit=True), + _bonus_rc(b, 0.0150, 0.7, 0), + _bonus_rc(c, 0.0150, 0.6, 0), + _bonus_rc(a, 0.0120, 0.5, 1), + _bonus_rc(d, 0.0100, 0.1, 0), + ] + + +#: The golden document order the OLD pre-phase loop (stable score- +#: descending walk, first-seen-chunk dedupe) produces over +#: :func:`_bonus_chunks` — pinned byte-identical by the kill switch. +GOLDEN_PRE_PHASE_ORDER = ["a.md", "gitea/README.md", "b.md", "c.md", "d.md"] + + +def test_selection_order_bonus_zero_is_the_pre_phase_golden_walk() -> None: + """LOCKED A3 kill switch: ``bonus=0`` returns the EXACT pre-phase + document order of the old score-descending first-seen walk — the + golden list pinned from the old loop over the mixed fused list + (incl. the b/c fused-score tie resolved by the input order the + fusion produced — the walk never re-sorts it away).""" + out = _selection_order(_bonus_chunks(), 0.0) + assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER + # The re-rank inputs are exposed and exact: effective == best fused + # score (no bonus), best cosine tracked across a doc's chunks (a: 0.9 + # from its rank-1 chunk, not 0.5), first-seen index in the + # score-descending walk. + assert [eff for _d, eff, _cos, _idx in out] == [ + 0.0200, 0.0160, 0.0150, 0.0150, 0.0100 + ] + assert [cos for _d, _eff, cos, _idx in out] == [ + pytest.approx(v) for v in (0.9, 0.0, 0.7, 0.6, 0.1) + ] + assert [idx for _d, _eff, _cos, idx in out] == [0, 1, 2, 3, 5] + + +def test_selection_order_bonus_inert_without_name_hits() -> None: + """No name-hit chunk present → the bonus cannot fire: the order is + IDENTICAL to the pre-phase walk even with the default bonus on + (LOCKED A3).""" + chunks = _bonus_chunks() + for rc in chunks: + rc.name_hit = False + out = _selection_order(chunks, 0.005) + assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER + + +def test_selection_order_bonus_lifts_name_hit_doc_below_bonus_gap() -> None: + """The name-hit doc's effective 0.016 + 0.005 = 0.021 EXCEEDS a's + 0.020 — a gap of 0.004 < bonus 0.005 — so the bonus lifts it to + rank 1; the rest keep their fused order (the bonus re-ranks, it + does not inflate).""" + out = _selection_order(_bonus_chunks(), 0.005) + assert [d.path for d, _eff, _cos, _idx in out] == [ + "gitea/README.md", "a.md", "b.md", "c.md", "d.md" + ] + assert out[0][1] == pytest.approx(0.016 + 0.005) + + +def test_selection_order_bonus_does_not_lift_above_bonus_gap() -> None: + """A gap LARGER than the bonus is not closed: the name-hit doc's + best 0.010 + 0.005 = 0.015 ties b/c on effective and LOSES to both + on the (−effective, −best_cosine) tie-break (its D1 cosine is 0.0) — + a keeps the lead (0.020). A second name-hit doc (``e.md``, also + 0.010/cos 0.0) trails gitea on the ``document.path`` tie-break — + the full re-rank key pinned.""" + chunks = _bonus_chunks() + chunks[1].score = 0.010 # the name-hit doc drops to a 0.010 best + e = _doc("e.md", "E" * 50) + chunks.insert(2, _bonus_rc(e, 0.010, 0.0, 0, name_hit=True)) + out = _selection_order(chunks, 0.005) + assert [d.path for d, _eff, _cos, _idx in out] == [ + "a.md", "b.md", "c.md", "e.md", "gitea/README.md", "d.md" + ] + + +def test_selection_order_first_seen_breaks_equal_path_ties() -> None: + """Two documents sharing a path across sources (``notes.md`` in two + sources) can tie on (effective, cosine, path) — the pre-bonus + first-seen rank decides (the last key element).""" + s1 = _doc("notes.md", "X" * 50, source="Src1") + s2 = _doc("notes.md", "Y" * 50, source="Src2") + chunks = [ + _bonus_rc(s1, 0.016, 0.0, 0, name_hit=True), + _bonus_rc(s2, 0.016, 0.0, 0, name_hit=True), + ] + out = _selection_order(chunks, 0.005) + assert [d.source for d, *_ in out] == ["Src1", "Src2"] + + +def test_selection_order_bonus_applied_once_per_document() -> None: + """The bonus is per DOCUMENT — applied ONCE no matter how many of + the doc's chunks are name hits (3×bonus would push the name-hit doc + above the 0.030 leader; one bonus cannot).""" + gitea = _doc("gitea/README.md", "G" * 50) + a = _doc("a.md", "A" * 50) + chunks = [ + _bonus_rc(a, 0.030, 0.8), + _bonus_rc(gitea, 0.016, 0.0, 0, name_hit=True), + _bonus_rc(gitea, 0.010, 0.0, 1, name_hit=True), + _bonus_rc(gitea, 0.008, 0.0, 2, name_hit=True), + ] + out = _selection_order(chunks, 0.005) + assert [d.path for d, _eff, _cos, _idx in out] == ["a.md", "gitea/README.md"] + assert out[1][1] == pytest.approx(0.016 + 0.005) # best + ONE bonus + + +def test_selection_order_bonus_fires_when_name_hit_is_not_first_chunk() -> None: + """The bonus fires on ANY name-hit chunk of the document — including + when the doc's first-seen (best) chunk is an ordinary vector row and + only a lower-ranked chunk is the D1 name-hit representative (a + document's name hit and its best chunk can be different chunks). + The bonus still lands on the doc's BEST fused score, and the doc's + best cosine stays tracked across ALL its chunks.""" + gitea = _doc("gitea/README.md", "G" * 50) + a = _doc("a.md", "A" * 50) + chunks = [ + _bonus_rc(a, 0.024, 0.8), + _bonus_rc(gitea, 0.020, 0.2, 0), # the doc's best — an ordinary chunk + _bonus_rc(gitea, 0.016, 0.55, 2, name_hit=True), # the name-hit rep + ] + assert [d.path for d, *_ in _selection_order(chunks, 0.0)] == [ + "a.md", "gitea/README.md" + ] + out = _selection_order(chunks, 0.005) + assert [d.path for d, *_ in out] == ["gitea/README.md", "a.md"] + assert out[0][1] == pytest.approx(0.020 + 0.005) # bonus on the BEST score + # The doc's best cosine is tracked across ALL its chunks — the + # lower-ranked name-hit chunk (0.55) beats the first-seen chunk's + # 0.2 (the re-rank tie-break input). + assert out[0][2] == pytest.approx(0.55) + + +def test_suggested_bonus_default_from_settings_and_kill_switch( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The *bonus* parameter defaults to the LIVE ``BOR_NAME_HIT_BONUS`` + setting (the ``n`` parameter's settings-read pattern — the default + 0.005 is the production value, not a frozen constant); an explicit + ``bonus=0`` and a settings kill switch both reproduce the pre-phase + golden walk (LOCKED A3).""" + chunks = _bonus_chunks() + settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue] + assert settings.name_hit_bonus == 0.005 # the production default + monkeypatch.setattr(retriever, "get_settings", lambda: settings) + assert [d.path for d in select_suggested(chunks, n=5)] == [ + "gitea/README.md", "a.md", "b.md", "c.md", "d.md" + ] + # Explicit kill switch: the byte-identical pre-phase order. + assert [d.path for d in select_suggested(chunks, n=5, bonus=0.0)] == GOLDEN_PRE_PHASE_ORDER + # Settings kill switch (BOR_NAME_HIT_BONUS=0) — the same golden walk. + off = Settings(_env_file=None, name_hit_bonus=0.0) # pyright: ignore[reportCallIssue] + monkeypatch.setattr(retriever, "get_settings", lambda: off) + assert [d.path for d in select_suggested(chunks, n=5)] == GOLDEN_PRE_PHASE_ORDER + + +def test_related_skips_excluded_ids_under_the_bonus() -> None: + """Exclusion is orthogonal to the bonus: excluded ids are skipped + exactly as before, on the bonus-adjusted walk — an excluded doc + never rides the related row even when the bonus would lift it to + the lead.""" + chunks = _bonus_chunks() + b_id = chunks[2].document.id + d_id = chunks[5].document.id + out = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.005) + assert [d.path for d in out] == ["gitea/README.md", "a.md", "c.md"] + # Kill switch: the same exclusions on the pre-phase walk. + out0 = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.0) + assert [d.path for d in out0] == ["a.md", "gitea/README.md", "c.md"] + # The name-hit doc itself excluded → the lead goes to the next doc. + g_id = chunks[1].document.id + out2 = select_related(chunks, {g_id}, cap=5, bonus=0.005) + assert [d.path for d in out2][0] == "a.md" + + +def test_weak_hit_titles_bonus_adjusted_order() -> None: + """Titles follow the bonus-adjusted selection walk (``_doc`` titles + equal paths here, so the title list mirrors the doc order); the + kill switch returns the pre-phase golden order.""" + chunks = _bonus_chunks() + assert weak_hit_titles(chunks, bonus=0.005) == [ + "gitea/README.md", "a.md", "b.md", "c.md", "d.md" + ] + assert weak_hit_titles(chunks, bonus=0.0) == GOLDEN_PRE_PHASE_ORDER + + +def test_bonus_lives_in_the_selection_layer_only() -> None: + """LOCKED A3: the bonus never touches the chunk objects — + ``score``/``cosine``/``fts_hit`` (the A8 gate's inputs — + ``query_log.top_score`` is the best fused chunk score, the same + values) are unchanged after every selection walk, even with the + bonus lifting a document.""" + chunks = _bonus_chunks() + before = { + rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks + } + select_suggested(chunks, n=5, bonus=0.005) + select_related(chunks, set(), cap=5, bonus=0.005) + weak_hit_titles(chunks, bonus=0.005) + after = { + rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks + } + assert before == after diff --git a/tests/unit/test_source_chip_quality.py b/tests/unit/test_source_chip_quality.py index 7c7f0fe..72a12ef 100644 --- a/tests/unit/test_source_chip_quality.py +++ b/tests/unit/test_source_chip_quality.py @@ -633,6 +633,13 @@ class _FakeSession: def get(self, _model: Any, _pk: Any) -> Any: return None + def execute(self, *args: Any, **kwargs: Any) -> list[Any]: + # Phase 119 (D3): the endpoint's suggested-folder lines run the + # ls catalog fetchers on this session — the fake catalog is + # empty (header-only lines), keeping the prompt builds + # deterministic here. + return [] + @pytest.fixture(autouse=True) def _admin_signed_in(client: TestClient) -> None: @@ -736,17 +743,19 @@ def test_shape_4_meta_question_deflected_frame_has_no_chips_or_row( assert "108_history_wire_check/00_phase.md" in row.sources -def test_done_frame_carries_suggested_refs_strong_plus_weak( +def test_done_frame_zero_read_grounded_turn_carries_no_chips( client: TestClient, chip_env: tuple[_FakeSession, _CannedLLM], monkeypatch: pytest.MonkeyPatch, ) -> None: - """Shape 2 on the wire under phase 118 (LOCKED A4): the citation - surface is the suggested tier + the agent's reads (deduped) — - with two retrieved docs and no read, BOTH docs are ``sources`` - refs (no floor — A3); nothing reaches rank 6+, so ``related`` is - empty; the tiers stay disjoint (the done frame's dedupe). The - durable record keeps the FULL retrieval (LOCKED A3).""" + """Shape 2 on the wire under phase 119 (LOCKED A1 — the phase-118 + A4 union retired): the citation surface is the agent's READ DOCS + only — with two retrieved docs and no read, ``sources`` is empty + (the zero-read grounded turn chips nothing — an accepted, + owner-directed consequence); the never-read suggested docs appear + NOWHERE in the frame (the explicit contrast against the retired + A4 union); nothing reaches rank 6+, so ``related`` is empty. The + durable record keeps the FULL retrieval (118-A3 stands).""" session, _llm = chip_env validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh", "VALIDATE_SH_CONTENT") @@ -766,14 +775,11 @@ def test_done_frame_carries_suggested_refs_strong_plus_weak( frames = _ask(client, "How does the phase gate decide to validate?") done = frames[-1] assert done["deflected"] is False - assert [(s["source"], s["path"]) for s in done["sources"]] == [ - ("brain-of-reese", ".agents/validate.sh"), - ("ServMon", "README.md"), # A4: suggested + read — both suggested (A3) - ] + # A1: read docs only — the agent read nothing ⇒ no chips, and the + # never-read suggested docs appear NOWHERE in the frame (the retired + # A4 union would have carried both). + assert done["sources"] == [] assert done["related"] == [] # nothing reaches rank 6+ for 2 docs - cited = {(s["source"], s["path"]) for s in done["sources"]} - related = {(s["source"], s["path"]) for s in done["related"]} - assert cited.isdisjoint(related) (row,) = session.added assert isinstance(row, QueryLog) @@ -788,14 +794,15 @@ def test_agent_read_related_doc_is_cited_not_related( chip_env: tuple[_FakeSession, _CannedLLM], monkeypatch: pytest.MonkeyPatch, ) -> None: - """The agent-read exemption (LOCKED A4, phase-118 tiering): a - rank-6+ doc — the related tier ("nearby docs") — that the agent - ``read`` via the tool is cited by definition: the model read it, so - it was used. It joins ``sources`` (after the suggested docs — it - was not suggested, so the read appends it last, deduped) and is - EXCLUDED from ``related`` (a used doc must never read as "nearby"); - the other rank-6+ doc stays in the tier. The read content reached - the model (the tool result in the follow-up request).""" + """The agent-read exemption (phase-119 A1 × phase-118 tiering): + ``sources`` is the READ DOCS only — a rank-6+ doc (the related + tier, "nearby docs") that the agent ``read`` via the tool is the + frame's ONLY chip: the model read it, so it was used. It is EXCLUDED + from ``related`` (a used doc must never read as "nearby" — unchanged + intent, the dedupe now keyed on read docs); the other rank-6+ doc + stays in the tier; the never-read suggested docs appear NOWHERE in + the frame (contrast against the retired A4 union). The read content + reached the model (the tool result in the follow-up request).""" session, _default_llm = chip_env strong = _doc("docs", "strong.md", "Strong", "STRONG_DOC_CONTENT") fillers = [ @@ -838,10 +845,10 @@ def test_agent_read_related_doc_is_cited_not_related( assert "WEAK_B_READ_BY_AGENT" in tool_msgs[0]["content"] sources = [(s["source"], s["path"]) for s in done["sources"]] - # A4: suggested (5) + the read doc (last — it was not suggested). - assert sources[-1] == ("docs", "weak-b.md") # read ⇒ cited, last - assert len(sources) == 6 - assert ("docs", "weak-c.md") not in sources # never suggested, never read + # A1: read docs only — the read related doc is the ONLY chip (the + # never-read suggested docs are nowhere in the frame). + assert sources == [("docs", "weak-b.md")] # read ⇒ chip + assert ("docs", "weak-c.md") not in sources # never read related = [(s["source"], s["path"]) for s in done["related"]] assert related == [("docs", "weak-c.md")] # the read doc is not "nearby" assert set(sources).isdisjoint(set(related))