phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -10,11 +10,15 @@ deterministic mock embeddings (same pattern as the earlier story suites).
|
||||
The four tests map the story's acceptance criteria:
|
||||
|
||||
1. multi-format fixture import — hidden doc excluded, ``/api/docs`` counts
|
||||
2. "How did I install gitlab?" — grounded (not deflected), gitlab chip,
|
||||
``query_log`` row with the gitlab doc in ``sources``
|
||||
2. "How did I install gitlab?" — grounded (not deflected), ZERO citation
|
||||
chips (phase 119, LOCKED A1 — the turn reads nothing: the suggested
|
||||
gitlab doc is context, not a chip), ``query_log`` row with the gitlab
|
||||
doc in ``sources`` (LOCKED A3, untouched)
|
||||
3. keyword-only question ("kafkabridge") beats the vector ranking — the
|
||||
corroborated-lexical gate (A8 revised 2026-09-14) grounds it end to
|
||||
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor
|
||||
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor;
|
||||
the FTS-matched doc tops the durable record (the chip pin is retired
|
||||
with the phase-118 A4 union — a zero-read turn chips nothing)
|
||||
4. "sourdough" — deflected bubble + ≥2 "Maybe try" chips
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -156,9 +160,15 @@ def test_gitlab_question_is_grounded_with_gitlab_chip(
|
||||
# Grounded: no deflected bubble at all.
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# The gitlab document is cited (a chip carrying its path).
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="container_gitlab/gitlab.md")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
|
||||
# chips — the suggested gitlab doc is seed context, not a citation
|
||||
# (the retired phase-118 A4 union is gone); the ranking assertion
|
||||
# lives in the durable record below (LOCKED A3, untouched). The
|
||||
# button label is the settle sync: the client re-labels Send on the
|
||||
# done frame, and the server writes the query_log row just before
|
||||
# yielding it.
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
# Durable record: not deflected, and the gitlab doc is in sources.
|
||||
with SessionLocal() as db:
|
||||
@@ -186,10 +196,15 @@ def test_keyword_only_question_beats_vector_ranking(
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# The FTS-matched doc is the TOP source chip (it beats the vector rank).
|
||||
first_chip = page.locator(".msg.brain .source-chip").first
|
||||
first_chip.wait_for(state="visible", timeout=30_000)
|
||||
expect(first_chip).to_contain_text("static-dns.json")
|
||||
# Phase 119 (LOCKED A1): zero-read turn → ZERO chips (the
|
||||
# FTS-matched doc beats the vector rank in the DURABLE record below
|
||||
# — LOCKED A3, untouched; the retired phase-118 A4 chip pin is
|
||||
# gone). Wait for the settle via the Send label instead of a chip
|
||||
# (the in-flight button is the enabled Stop control — only the
|
||||
# label proves the done frame landed and the query_log row was
|
||||
# written).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
with SessionLocal() as db:
|
||||
row = db.scalars(select(QueryLog)).one()
|
||||
|
||||
Reference in New Issue
Block a user