phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -1,22 +1,27 @@
|
||||
"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface
|
||||
contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract
|
||||
replaced the phase-112/113 usefulness bar: the chip row is the
|
||||
SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on
|
||||
these turns), the related row is rank 6+ (capped at
|
||||
``related_max_docs`` = 2), and a deflected turn still cites nothing
|
||||
(done.sources = [] — its weak hits are suggested for the durable record
|
||||
but are never citation chips).
|
||||
"""Phase 113 E2E (Playwright) — phase 118 re-targeted, phase 119
|
||||
re-targeted again: the citation-surface contract (phase 119, LOCKED
|
||||
A1) as VISIBLE chip counts — chips are the AGENT-READ docs only (the
|
||||
retired phase-118 A4 suggested+read union is gone): neither turn on
|
||||
this suite reads anything, so a grounded turn shows ZERO citation
|
||||
chips (the suggested tier is seed context, not citations — the
|
||||
phase-112/113 usefulness bar was retired with the full-text seeds, and
|
||||
the suggested chips were retired with the phase-118 A4 union), the
|
||||
related row is rank 6+ (capped at ``related_max_docs`` = 2, untouched),
|
||||
and a deflected turn still cites nothing (done.sources = [] — its weak
|
||||
hits are suggested for the durable record but are never citation
|
||||
chips).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov
|
||||
|
||||
Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row
|
||||
is the suggested tier — the 5 docs the model was seeded with (its
|
||||
"start here" set — the old "one chip per bar-clearing doc" is retired
|
||||
with the full-text seeds; the LLM decides what the summaries earn);
|
||||
a deflected turn shows zero citation chips (its weak hits, if any, live
|
||||
in the de-emphasized related row — ``.related-doc`` links, NEVER
|
||||
Acceptance (TODO L144–146, phase-119 shape): a grounded turn's chip row
|
||||
is the READ set — empty on these no-read turns (the visible consequence
|
||||
of the owner decision 2026-09-16: the suggested docs the model was
|
||||
seeded with are context, not citations — the LLM decides what the
|
||||
summaries earn, and a zero-read turn chips nothing); a deflected turn
|
||||
shows zero citation chips (its weak hits, if any, live in the
|
||||
de-emphasized related row — ``.related-doc`` links, NEVER
|
||||
``.source-chip``; L2c: never render weak hits as answer citations).
|
||||
|
||||
The fixture KB's tier shapes are deterministic under the E2E mock's
|
||||
@@ -25,14 +30,20 @@ here against the wire):
|
||||
|
||||
* **grounded question** — "What SSH aliases do I have?": best cosine
|
||||
0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is
|
||||
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md
|
||||
> backups.md (fused rank — the lexical-only cosine-0.0 docs rank when
|
||||
they rank: no floor filters them, LOCKED A3) and the done frame
|
||||
carries EXACTLY those five cited refs; the rank-6+ remainder
|
||||
(compose.container, static-dns.json) rides the related row, capped at
|
||||
two. (The four OBSERVED live shapes are unit-pinned at plan level in
|
||||
``tests/unit/test_source_chip_quality.py`` — the fixture KB
|
||||
reproduces the tiered shape live, so no docstring caveat is needed.)
|
||||
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > backups.md
|
||||
> kubernetes.md (fused rank — the lexical-only cosine-0.0 docs rank
|
||||
when they rank: no floor filters them, LOCKED A3; phase 119, D1 —
|
||||
``aliases`` is a stem sub-component of ``ssh_aliases.txt``, so the
|
||||
doc name-hits, its name-hit row LEADS the lexical list, and the FTS
|
||||
rank shift re-orders the fused scores — the pre-phase
|
||||
kubernetes-before-backups order is retired) and the done frame
|
||||
carries NO cited refs (phase 119 A1 — nothing was read: the
|
||||
suggested tier seeds the prompt but never chips); the rank-6+
|
||||
remainder (compose.container, static-dns.json) rides the related
|
||||
row, capped at two. (The four OBSERVED live shapes are unit-pinned at
|
||||
plan level in ``tests/unit/test_source_chip_quality.py`` — the
|
||||
fixture KB reproduces the tiered shape live, so no docstring caveat
|
||||
is needed.)
|
||||
* **deflected question** — "How do I bake sourdough bread?": best cosine
|
||||
0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs
|
||||
(done.sources = []); the weak hits ARE suggested (no floor) for the
|
||||
@@ -106,14 +117,15 @@ def _ask(page: Page, message: str) -> None:
|
||||
def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""The acceptance pin (phase 118, A4): a grounded question → the done
|
||||
bubble carries EXACTLY the suggested tier as ``.source-chip``s (the
|
||||
five docs the model was seeded with — top-5, NO floor: the old
|
||||
usefulness-bar "one chip" is retired with the full-text seeds) and
|
||||
the rank-6+ remainder renders only in the de-emphasized
|
||||
``.related-docs`` row (``.related-doc`` links — never
|
||||
``.source-chip``), labeled "Nearby docs, in case:". The durable
|
||||
record keeps the FULL retrieval (LOCKED A3)."""
|
||||
"""The acceptance pin (phase 119, A1): a grounded, NO-READ question →
|
||||
the done bubble carries ZERO ``.source-chip``s (the chip row is the
|
||||
agent-read docs only — nothing was read here: the five suggested
|
||||
docs the model was seeded with are context, not citations — the
|
||||
retired phase-118 A4 union is gone) and the rank-6+ remainder
|
||||
renders only in the de-emphasized ``.related-docs`` row
|
||||
(``.related-doc`` links — never ``.source-chip``), labeled "Nearby
|
||||
docs, in case:" (phase-113 behavior, untouched). The durable record
|
||||
keeps the FULL retrieval (LOCKED A3, untouched)."""
|
||||
_reset_db(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
@@ -127,16 +139,14 @@ def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
# Grounded: no deflected bubble at all.
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# The citation surface IS the suggested tier (LOCKED A4 — top-5,
|
||||
# NO floor): five chips, in fused rank order — the model was
|
||||
# seeded with exactly these five docs' summaries.
|
||||
chip = page.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(5, timeout=30_000)
|
||||
expect(chip.nth(0)).to_contain_text("ssh_aliases.txt")
|
||||
expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml")
|
||||
expect(chip.nth(2)).to_contain_text("uptime_probe.py")
|
||||
expect(chip.nth(3)).to_contain_text("kubernetes.md")
|
||||
expect(chip.nth(4)).to_contain_text("backups.md")
|
||||
# Phase 119 (LOCKED A1): the citation surface is the READ docs only
|
||||
# — this turn read nothing, so ZERO chips; the five suggested docs
|
||||
# (ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py >
|
||||
# backups.md > kubernetes.md — the phase-119 D1 re-rank, pinned in
|
||||
# the durable record below) seeded the prompt but never chip —
|
||||
# the retired phase-118 A4 union is gone. The suggested tier itself
|
||||
# is pinned below by the durable record (LOCKED A3, untouched).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# The rank-6+ remainder rides the related row: a labeled,
|
||||
# de-emphasized list — one .related-doc link per doc (rank order,
|
||||
@@ -162,13 +172,19 @@ def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
row_log = db.scalars(select(QueryLog)).one()
|
||||
assert row_log.question == SINGLE_SOURCE_QUESTION
|
||||
assert row_log.deflected is False
|
||||
# Suggested tier + related remainder, in the logged order.
|
||||
# Suggested tier + related remainder, in the logged order. Phase
|
||||
# 119 (D1): the stem sub-component ``aliases`` name-hits
|
||||
# ``ssh_aliases.txt`` — its name-hit row leads the lexical list,
|
||||
# the FTS rank shift re-orders the fused scores, and ``backups.md``
|
||||
# (best eff 0.016277) now out-ranks ``kubernetes.md`` (0.016036);
|
||||
# re-pinned against the post-phase walk (the record's CONTENT —
|
||||
# suggested + related + read, deduped — is LOCKED A3, untouched).
|
||||
assert row_log.sources == (
|
||||
"docs/homelab/ssh/ssh_aliases.txt, "
|
||||
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
|
||||
"docs/homelab/scripts/uptime_probe.py, "
|
||||
"docs/homelab/kubernetes.md, "
|
||||
"docs/homelab/backups.md, "
|
||||
"docs/homelab/kubernetes.md, "
|
||||
"docs/homelab/quadlet/compose.container, "
|
||||
"docs/homelab/networking/static-dns.json"
|
||||
), row_log.sources
|
||||
|
||||
Reference in New Issue
Block a user