phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -59,19 +59,21 @@ TODO item — one Playwright file per story, A16):
|
||||
The inverse of the retired phase-24 pin: NO document's tail
|
||||
sentinel is in the echoed context — the full content of no
|
||||
suggested doc ever reached the model (case b). The grounded turn
|
||||
cites the 5 suggested docs (case e's chip surface, no read yet) and
|
||||
cites NOTHING (phase 119, LOCKED A1 — a zero-read turn chips
|
||||
nothing; the retired phase-118 A4 suggested+read union is gone) and
|
||||
renders the de-emphasized related row (rank 6+); the durable
|
||||
record carries suggested + related.
|
||||
record carries suggested + related (118-A3 untouched).
|
||||
2. ``test_read_suggested_doc_adds_full_text_and_cites`` — cases (c) +
|
||||
(e): the scripted flow ``read``s the rank-1 suggested doc — the
|
||||
read SUCCEEDS (phase 118: the seeds are summaries, not full text —
|
||||
the retired top-2 seed-read refusal is gone) and the mock's
|
||||
verbatim echo of the read result lands the tail sentinel in the
|
||||
answer (case c: the full text now arrives through the ``read``
|
||||
tool, not the seed); the citation chips = suggested + read,
|
||||
deduped — the read doc is among the suggested, so the chip row is
|
||||
exactly the 5 suggested docs, and the related row renders rank 6+
|
||||
(case e); the durable record carries suggested + related + read.
|
||||
tool, not the seed); the citation chips = the READ docs only
|
||||
(phase 119, LOCKED A1 — exactly the one read doc; none of the
|
||||
four other suggested docs chips), and the related row renders
|
||||
rank 6+ (case e); the durable record carries suggested + related +
|
||||
read (118-A3 untouched).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -503,14 +505,16 @@ def _last_query_log() -> QueryLog:
|
||||
return rows[0]
|
||||
|
||||
|
||||
def _assert_suggested_chips_and_related_row(page: Page, suggested_paths: list[str]) -> None:
|
||||
"""The citation surface of a grounded turn (LOCKED A4): the chip row
|
||||
is the suggested set (+ agent reads, deduped — asserted per test)
|
||||
and the de-emphasized ``related-docs`` row carries the rank-6+
|
||||
remainder (phase-113 UI, never a citation chip)."""
|
||||
def _assert_read_chips_and_related_row(page: Page, read_paths: list[str]) -> None:
|
||||
"""The citation surface of a grounded turn (phase 119, LOCKED A1 —
|
||||
the retired phase-118 A4 union is gone): the chip row is the
|
||||
AGENT-READ docs only (deduped, read order — a zero-read turn chips
|
||||
NOTHING: the suggested docs are seed context, not citations), and
|
||||
the de-emphasized ``related-docs`` row carries the rank-6+ remainder
|
||||
(phase-113 UI, untouched — never a citation chip)."""
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(len(suggested_paths))
|
||||
for path in suggested_paths:
|
||||
expect(chips).to_have_count(len(read_paths))
|
||||
for path in read_paths:
|
||||
expect(
|
||||
chips.filter(has_text=path), message=f"chip for {path}"
|
||||
).to_have_count(1)
|
||||
@@ -553,11 +557,12 @@ def test_summaries_seed_the_prompt_not_the_full_text(
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The grounded turn cites the 5 suggested docs (LOCKED A4 — no read
|
||||
# yet: the chip row IS the suggested set, in rank order).
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
] == [(SOURCE, p) for p in SUGGESTED_TAIL], done["sources"]
|
||||
# Phase 119 (LOCKED A1): the grounded done frame cites the READ docs
|
||||
# only — this turn read NOTHING (the summary-only fast path), so the
|
||||
# citation surface is empty; the suggested set is seed context, not
|
||||
# citations (the retired phase-118 A4 union is gone). The durable
|
||||
# record below still carries suggested + related (118-A3 untouched).
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
@@ -583,12 +588,13 @@ def test_summaries_seed_the_prompt_not_the_full_text(
|
||||
expect(bubble).not_to_contain_text(sentinel)
|
||||
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||
|
||||
# Case (e)'s chip surface (no read yet): chips = the 5 suggested
|
||||
# docs; the related row renders rank 6+ (the de-emphasized row).
|
||||
_assert_suggested_chips_and_related_row(page, SUGGESTED_TAIL)
|
||||
# Case (e)'s chip surface (phase 119 A1 — no read yet): ZERO chips;
|
||||
# the related row renders rank 6+ (the de-emphasized row, untouched).
|
||||
_assert_read_chips_and_related_row(page, [])
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3 — the log
|
||||
# records retrieval, not citations).
|
||||
# records retrieval, not citations; phase 119 A1 retires the chip
|
||||
# union only, not the record).
|
||||
row = _last_query_log()
|
||||
assert row.question == TAIL_QUESTION
|
||||
assert row.deflected is False
|
||||
@@ -609,10 +615,10 @@ def test_read_suggested_doc_adds_full_text_and_cites(
|
||||
retired top-2 seed-read refusal is gone) and the mock's verbatim
|
||||
echo of the read result lands the document's tail sentinel in the
|
||||
answer (case c: the full text reached the model through the
|
||||
``read`` tool, not the seed). The citation chips = suggested +
|
||||
read, deduped — the read doc is among the suggested, so the chip
|
||||
row is exactly the 5 suggested docs; the related row renders rank
|
||||
6+ (case e)."""
|
||||
``read`` tool, not the seed). The citation chips = the READ docs
|
||||
only (phase 119, LOCKED A1 — exactly the one read doc; none of the
|
||||
other four suggested docs chips; the retired phase-118 A4 union is
|
||||
gone); the related row renders rank 6+ (case e, untouched)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
@@ -628,13 +634,15 @@ def test_read_suggested_doc_adds_full_text_and_cites(
|
||||
], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# LOCKED A4: done.sources = suggested + read, deduped — the read doc
|
||||
# is already among the suggested, so the citation surface is
|
||||
# exactly the 5 suggested docs (in rank order).
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
|
||||
# exactly the one scripted read; the other four suggested docs
|
||||
# never chip (the retired phase-118 A4 suggested+read union is
|
||||
# gone).
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
] == [(SOURCE, p) for p in SUGGESTED_READ], done["sources"]
|
||||
# done.related = the rank-6+ remainder (deduped against the cited).
|
||||
] == [(SOURCE, READ_TARGET)], done["sources"]
|
||||
# done.related = the rank-6+ remainder (deduped against the cited —
|
||||
# the read doc is rank 1, not in the remainder).
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["related"]
|
||||
] == [(SOURCE, p) for p in RELATED_PATHS], done["related"]
|
||||
@@ -658,13 +666,13 @@ def test_read_suggested_doc_adds_full_text_and_cites(
|
||||
if path != READ_TARGET:
|
||||
expect(bubble).not_to_contain_text(sentinel)
|
||||
|
||||
# Case (e): the UI chip row = suggested + read (deduped — the read
|
||||
# doc is among the suggested, so chips = the 5 suggested docs), and
|
||||
# the related row renders rank 6+.
|
||||
_assert_suggested_chips_and_related_row(page, SUGGESTED_READ)
|
||||
# Case (e): the UI chip row = the READ docs only (phase 119 A1 —
|
||||
# exactly the one read doc), and the related row renders rank 6+.
|
||||
_assert_read_chips_and_related_row(page, [READ_TARGET])
|
||||
|
||||
# Durable record: grounded; suggested + related + read (deduped,
|
||||
# LOCKED A3).
|
||||
# LOCKED A3 — phase 119 A1 retires the chip union only, not the
|
||||
# record).
|
||||
row = _last_query_log()
|
||||
assert row.question == READ_QUESTION
|
||||
assert row.deflected is False
|
||||
|
||||
Reference in New Issue
Block a user