phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 18s

All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
2026-09-16 15:50:48 -04:00
parent 795fb56425
commit a5b63f83ad
89 changed files with 4377 additions and 655 deletions
+58 -42
View File
@@ -1,22 +1,27 @@
"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface
contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract
replaced the phase-112/113 usefulness bar: the chip row is the
SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on
these turns), the related row is rank 6+ (capped at
``related_max_docs`` = 2), and a deflected turn still cites nothing
(done.sources = [] — its weak hits are suggested for the durable record
but are never citation chips).
"""Phase 113 E2E (Playwright) — phase 118 re-targeted, phase 119
re-targeted again: the citation-surface contract (phase 119, LOCKED
A1) as VISIBLE chip counts — chips are the AGENT-READ docs only (the
retired phase-118 A4 suggested+read union is gone): neither turn on
this suite reads anything, so a grounded turn shows ZERO citation
chips (the suggested tier is seed context, not citations — the
phase-112/113 usefulness bar was retired with the full-text seeds, and
the suggested chips were retired with the phase-118 A4 union), the
related row is rank 6+ (capped at ``related_max_docs`` = 2, untouched),
and a deflected turn still cites nothing (done.sources = [] — its weak
hits are suggested for the durable record but are never citation
chips).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov
Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row
is the suggested tier — the 5 docs the model was seeded with (its
"start here" set — the old "one chip per bar-clearing doc" is retired
with the full-text seeds; the LLM decides what the summaries earn);
a deflected turn shows zero citation chips (its weak hits, if any, live
in the de-emphasized related row — ``.related-doc`` links, NEVER
Acceptance (TODO L144–146, phase-119 shape): a grounded turn's chip row
is the READ set — empty on these no-read turns (the visible consequence
of the owner decision 2026-09-16: the suggested docs the model was
seeded with are context, not citations — the LLM decides what the
summaries earn, and a zero-read turn chips nothing); a deflected turn
shows zero citation chips (its weak hits, if any, live in the
de-emphasized related row — ``.related-doc`` links, NEVER
``.source-chip``; L2c: never render weak hits as answer citations).
The fixture KB's tier shapes are deterministic under the E2E mock's
@@ -25,14 +30,20 @@ here against the wire):
* **grounded question** — "What SSH aliases do I have?": best cosine
0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md
> backups.md (fused rank — the lexical-only cosine-0.0 docs rank when
they rank: no floor filters them, LOCKED A3) and the done frame
carries EXACTLY those five cited refs; the rank-6+ remainder
(compose.container, static-dns.json) rides the related row, capped at
two. (The four OBSERVED live shapes are unit-pinned at plan level in
``tests/unit/test_source_chip_quality.py`` — the fixture KB
reproduces the tiered shape live, so no docstring caveat is needed.)
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > backups.md
> kubernetes.md (fused rank — the lexical-only cosine-0.0 docs rank
when they rank: no floor filters them, LOCKED A3; phase 119, D1 —
``aliases`` is a stem sub-component of ``ssh_aliases.txt``, so the
doc name-hits, its name-hit row LEADS the lexical list, and the FTS
rank shift re-orders the fused scores — the pre-phase
kubernetes-before-backups order is retired) and the done frame
carries NO cited refs (phase 119 A1 — nothing was read: the
suggested tier seeds the prompt but never chips); the rank-6+
remainder (compose.container, static-dns.json) rides the related
row, capped at two. (The four OBSERVED live shapes are unit-pinned at
plan level in ``tests/unit/test_source_chip_quality.py`` — the
fixture KB reproduces the tiered shape live, so no docstring caveat
is needed.)
* **deflected question** — "How do I bake sourdough bread?": best cosine
0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs
(done.sources = []); the weak hits ARE suggested (no floor) for the
@@ -106,14 +117,15 @@ def _ask(page: Page, message: str) -> None:
def test_single_source_question_shows_exactly_one_citation_chip(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""The acceptance pin (phase 118, A4): a grounded question → the done
bubble carries EXACTLY the suggested tier as ``.source-chip``s (the
five docs the model was seeded with — top-5, NO floor: the old
usefulness-bar "one chip" is retired with the full-text seeds) and
the rank-6+ remainder renders only in the de-emphasized
``.related-docs`` row (``.related-doc`` links — never
``.source-chip``), labeled "Nearby docs, in case:". The durable
record keeps the FULL retrieval (LOCKED A3)."""
"""The acceptance pin (phase 119, A1): a grounded, NO-READ question →
the done bubble carries ZERO ``.source-chip``s (the chip row is the
agent-read docs only — nothing was read here: the five suggested
docs the model was seeded with are context, not citations — the
retired phase-118 A4 union is gone) and the rank-6+ remainder
renders only in the de-emphasized ``.related-docs`` row
(``.related-doc`` links — never ``.source-chip``), labeled "Nearby
docs, in case:" (phase-113 behavior, untouched). The durable record
keeps the FULL retrieval (LOCKED A3, untouched)."""
_reset_db(mock_llm)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -127,16 +139,14 @@ def test_single_source_question_shows_exactly_one_citation_chip(
# Grounded: no deflected bubble at all.
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# The citation surface IS the suggested tier (LOCKED A4 — top-5,
# NO floor): five chips, in fused rank order — the model was
# seeded with exactly these five docs' summaries.
chip = page.locator(".msg.brain .source-chip")
expect(chip).to_have_count(5, timeout=30_000)
expect(chip.nth(0)).to_contain_text("ssh_aliases.txt")
expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml")
expect(chip.nth(2)).to_contain_text("uptime_probe.py")
expect(chip.nth(3)).to_contain_text("kubernetes.md")
expect(chip.nth(4)).to_contain_text("backups.md")
# Phase 119 (LOCKED A1): the citation surface is the READ docs only
# — this turn read nothing, so ZERO chips; the five suggested docs
# (ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py >
# backups.md > kubernetes.md — the phase-119 D1 re-rank, pinned in
# the durable record below) seeded the prompt but never chip —
# the retired phase-118 A4 union is gone. The suggested tier itself
# is pinned below by the durable record (LOCKED A3, untouched).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# The rank-6+ remainder rides the related row: a labeled,
# de-emphasized list — one .related-doc link per doc (rank order,
@@ -162,13 +172,19 @@ def test_single_source_question_shows_exactly_one_citation_chip(
row_log = db.scalars(select(QueryLog)).one()
assert row_log.question == SINGLE_SOURCE_QUESTION
assert row_log.deflected is False
# Suggested tier + related remainder, in the logged order.
# Suggested tier + related remainder, in the logged order. Phase
# 119 (D1): the stem sub-component ``aliases`` name-hits
# ``ssh_aliases.txt`` — its name-hit row leads the lexical list,
# the FTS rank shift re-orders the fused scores, and ``backups.md``
# (best eff 0.016277) now out-ranks ``kubernetes.md`` (0.016036);
# re-pinned against the post-phase walk (the record's CONTENT —
# suggested + related + read, deduped — is LOCKED A3, untouched).
assert row_log.sources == (
"docs/homelab/ssh/ssh_aliases.txt, "
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
"docs/homelab/scripts/uptime_probe.py, "
"docs/homelab/kubernetes.md, "
"docs/homelab/backups.md, "
"docs/homelab/kubernetes.md, "
"docs/homelab/quadlet/compose.container, "
"docs/homelab/networking/static-dns.json"
), row_log.sources