phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
@@ -1,35 +1,44 @@
|
||||
"""Phase 113 E2E (Playwright): the source-chip quality contract (TODO L5 +
|
||||
L2c) — the usefulness bar + the de-emphasized related-docs row, as VISIBLE
|
||||
chip counts.
|
||||
"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface
|
||||
contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract
|
||||
replaced the phase-112/113 usefulness bar: the chip row is the
|
||||
SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on
|
||||
these turns), the related row is rank 6+ (capped at
|
||||
``related_max_docs`` = 2), and a deflected turn still cites nothing
|
||||
(done.sources = [] — its weak hits are suggested for the durable record
|
||||
but are never citation chips).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov
|
||||
|
||||
Acceptance (TODO L144–146): "for a single-document question, the turn
|
||||
shows one citation chip"; a weak 2nd doc renders only in the
|
||||
de-emphasized related row (``.related-doc`` links, NEVER ``.source-chip``);
|
||||
a deflected turn shows zero citation chips (its weak hits, if any, live in
|
||||
the related row).
|
||||
Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row
|
||||
is the suggested tier — the 5 docs the model was seeded with (its
|
||||
"start here" set — the old "one chip per bar-clearing doc" is retired
|
||||
with the full-text seeds; the LLM decides what the summaries earn);
|
||||
a deflected turn shows zero citation chips (its weak hits, if any, live
|
||||
in the de-emphasized related row — ``.related-doc`` links, NEVER
|
||||
``.source-chip``; L2c: never render weak hits as answer citations).
|
||||
|
||||
The fixture KB's tier shapes are deterministic under the E2E mock's
|
||||
bag-of-words embeddings + the mock-calibrated bar (conftest:
|
||||
``BOR_SOURCE_USEFULNESS_FLOOR=0.15``, half the 0.30 threshold — like the
|
||||
lexical floor):
|
||||
bag-of-words embeddings + the fused rank walk (probe-verified, pinned
|
||||
here against the wire):
|
||||
|
||||
* **single-source question** — "What SSH aliases do I have?":
|
||||
``ssh_aliases.txt`` is the ONLY doc whose best-chunk cosine clears the
|
||||
bar (0.352 ≥ 0.15; grounded at 0.352 ≥ 0.30) → the done frame carries
|
||||
exactly ONE cited ref; the two below-bar docs (gitlab-compose.yaml
|
||||
0.025, uptime_probe.py 0.113) ride the related tier. This IS the
|
||||
strong+weak two-tier shape on the wire (the four OBSERVED live shapes
|
||||
are unit-pinned at plan level in
|
||||
``tests/unit/test_source_chip_quality.py`` — the fixture KB reproduces
|
||||
the same shape live, so no docstring caveat is needed).
|
||||
* **grounded question** — "What SSH aliases do I have?": best cosine
|
||||
0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is
|
||||
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md
|
||||
> backups.md (fused rank — the lexical-only cosine-0.0 docs rank when
|
||||
they rank: no floor filters them, LOCKED A3) and the done frame
|
||||
carries EXACTLY those five cited refs; the rank-6+ remainder
|
||||
(compose.container, static-dns.json) rides the related row, capped at
|
||||
two. (The four OBSERVED live shapes are unit-pinned at plan level in
|
||||
``tests/unit/test_source_chip_quality.py`` — the fixture KB
|
||||
reproduces the tiered shape live, so no docstring caveat is needed.)
|
||||
* **deflected question** — "How do I bake sourdough bread?": best cosine
|
||||
0.109 < 0.30 and zero FTS hits → honest deflection; nothing clears the
|
||||
bar → zero cited refs; the weak hits (new-service.md 0.109,
|
||||
ssh_aliases.txt 0.050) ride the related row.
|
||||
0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs
|
||||
(done.sources = []); the weak hits ARE suggested (no floor) for the
|
||||
durable record — new-service.md > ssh_aliases.txt > lan.network >
|
||||
uptime_probe.py > compose.container — and the rank-6+ remainder
|
||||
(backups.md, kubernetes.md) rides the related row.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -97,12 +106,14 @@ def _ask(page: Page, message: str) -> None:
|
||||
def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""The acceptance pin: a single-document question → the done bubble
|
||||
carries EXACTLY ONE ``.source-chip`` (the bar-clearing doc) and the
|
||||
below-bar docs render only in the de-emphasized ``.related-docs`` row
|
||||
(``.related-doc`` links — never ``.source-chip``), labeled
|
||||
"Nearby docs, in case:". The durable record keeps the FULL retrieval
|
||||
(LOCKED A3)."""
|
||||
"""The acceptance pin (phase 118, A4): a grounded question → the done
|
||||
bubble carries EXACTLY the suggested tier as ``.source-chip``s (the
|
||||
five docs the model was seeded with — top-5, NO floor: the old
|
||||
usefulness-bar "one chip" is retired with the full-text seeds) and
|
||||
the rank-6+ remainder renders only in the de-emphasized
|
||||
``.related-docs`` row (``.related-doc`` links — never
|
||||
``.source-chip``), labeled "Nearby docs, in case:". The durable
|
||||
record keeps the FULL retrieval (LOCKED A3)."""
|
||||
_reset_db(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
@@ -116,52 +127,64 @@ def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
# Grounded: no deflected bubble at all.
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# EXACTLY ONE citation chip — the bar-clearing doc (the 2nd chip of
|
||||
# the pre-phase turn demoted to the row; the acceptance criterion).
|
||||
# The citation surface IS the suggested tier (LOCKED A4 — top-5,
|
||||
# NO floor): five chips, in fused rank order — the model was
|
||||
# seeded with exactly these five docs' summaries.
|
||||
chip = page.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
expect(chip.first).to_contain_text("ssh_aliases.txt")
|
||||
expect(chip).to_have_count(5, timeout=30_000)
|
||||
expect(chip.nth(0)).to_contain_text("ssh_aliases.txt")
|
||||
expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml")
|
||||
expect(chip.nth(2)).to_contain_text("uptime_probe.py")
|
||||
expect(chip.nth(3)).to_contain_text("kubernetes.md")
|
||||
expect(chip.nth(4)).to_contain_text("backups.md")
|
||||
|
||||
# The below-bar docs ride the related row: a labeled, de-emphasized
|
||||
# list — one .related-doc link per doc (rank order, capped at
|
||||
# related_max_docs = 2), never a .source-chip.
|
||||
# The rank-6+ remainder rides the related row: a labeled,
|
||||
# de-emphasized list — one .related-doc link per doc (rank order,
|
||||
# capped at related_max_docs = 2), never a .source-chip.
|
||||
row = page.locator(".msg.brain .related-docs")
|
||||
expect(row).to_have_count(1)
|
||||
expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case")
|
||||
expect(row.first.locator(".related-docs-label")).to_have_text("Nearby docs, in case:")
|
||||
links = page.locator(".msg.brain .related-docs .related-doc")
|
||||
expect(links).to_have_count(2, timeout=30_000)
|
||||
expect(links.nth(0)).to_contain_text("gitlab-compose.yaml")
|
||||
expect(links.nth(1)).to_contain_text("uptime_probe.py")
|
||||
expect(links.nth(0)).to_contain_text("compose.container")
|
||||
expect(links.nth(1)).to_contain_text("static-dns.json")
|
||||
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
|
||||
# The related links keep the chip's /document.html href + identity.
|
||||
expect(links.first).to_have_attribute(
|
||||
"title", "docs/homelab/container_gitlab/gitlab-compose.yaml"
|
||||
"title", "docs/homelab/quadlet/compose.container"
|
||||
)
|
||||
|
||||
# Durable record: not deflected; the FULL retrieval (cited + related)
|
||||
# is logged — query_log records retrieval, not citations (LOCKED A3).
|
||||
# Durable record: not deflected; the FULL retrieval (suggested +
|
||||
# related + read, deduped) is logged — query_log records retrieval,
|
||||
# not citations (LOCKED A3).
|
||||
with SessionLocal() as db:
|
||||
row_log = db.scalars(select(QueryLog)).one()
|
||||
assert row_log.question == SINGLE_SOURCE_QUESTION
|
||||
assert row_log.deflected is False
|
||||
for path in (
|
||||
"homelab/ssh/ssh_aliases.txt",
|
||||
"homelab/container_gitlab/gitlab-compose.yaml",
|
||||
"homelab/scripts/uptime_probe.py",
|
||||
):
|
||||
assert path in row_log.sources
|
||||
# Suggested tier + related remainder, in the logged order.
|
||||
assert row_log.sources == (
|
||||
"docs/homelab/ssh/ssh_aliases.txt, "
|
||||
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
|
||||
"docs/homelab/scripts/uptime_probe.py, "
|
||||
"docs/homelab/kubernetes.md, "
|
||||
"docs/homelab/backups.md, "
|
||||
"docs/homelab/quadlet/compose.container, "
|
||||
"docs/homelab/networking/static-dns.json"
|
||||
), row_log.sources
|
||||
|
||||
|
||||
def test_deflected_question_shows_zero_citation_chips(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""The acceptance pin: a deflected turn (known-out-of-KB) → ZERO
|
||||
``.source-chip`` elements under the bubble; the weak hits live in the
|
||||
related row only — its links are ``.related-doc``, never
|
||||
``.source-chip`` (L2c: "at minimum: never render them as answer
|
||||
citations"). The "Maybe try" chips and the durable record are
|
||||
unchanged."""
|
||||
"""The acceptance pin (phase 118, A4): a deflected turn
|
||||
(known-out-of-KB) → ZERO ``.source-chip`` elements under the bubble
|
||||
(done.sources = [] — a deflected answer cites nothing; the weak hits
|
||||
are suggested for the durable record but are scored docs, not
|
||||
citations); the rank-6+ weak hits live in the related row only —
|
||||
its links are ``.related-doc``, never ``.source-chip`` (L2c: "at
|
||||
minimum: never render them as answer citations"). The "Maybe try"
|
||||
chips are unchanged."""
|
||||
_reset_db(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
@@ -181,21 +204,35 @@ def test_deflected_question_shows_zero_citation_chips(
|
||||
row = page.locator(".msg.brain .related-docs")
|
||||
expect(row).to_have_count(1, timeout=30_000)
|
||||
links = page.locator(".msg.brain .related-docs .related-doc")
|
||||
expect(links).to_have_count(2) # the weak hits, rank order, capped at 2
|
||||
expect(links.nth(0)).to_contain_text("new-service.md")
|
||||
expect(links.nth(1)).to_contain_text("ssh_aliases.txt")
|
||||
# The rank-6+ remainder of the weak hits, rank order, capped at 2 —
|
||||
# the top five weak hits are suggested (no floor) for the durable
|
||||
# record, but a deflected turn cites nothing, so they never render
|
||||
# as chips.
|
||||
expect(links).to_have_count(2)
|
||||
expect(links.nth(0)).to_contain_text("backups.md")
|
||||
expect(links.nth(1)).to_contain_text("kubernetes.md")
|
||||
|
||||
# ZERO citation chips under the bubble — the weak hits are scored
|
||||
# docs, not citations (the phase-112/113 contract on the wire).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
|
||||
|
||||
# Durable record: deflected, weak top score — the retrieval stays
|
||||
# logged for threshold tuning (LOCKED A3: observability unchanged).
|
||||
# Durable record: deflected, weak top score — the FULL retrieval
|
||||
# (the weak hits ARE suggested — no floor — plus the rank-6+ related
|
||||
# remainder) stays logged for threshold tuning (LOCKED A3:
|
||||
# observability unchanged).
|
||||
with SessionLocal() as db:
|
||||
row_log = db.scalars(select(QueryLog)).one()
|
||||
assert row_log.question == OFF_TOPIC
|
||||
assert row_log.deflected is True
|
||||
assert 0.0 < row_log.top_score < get_settings().relevance_threshold
|
||||
assert row_log.fts_hits == 0
|
||||
assert row_log.sources # the weak-hit paths, for threshold tuning
|
||||
assert row_log.sources == (
|
||||
"docs/deployments/new-service.md, "
|
||||
"docs/homelab/ssh/ssh_aliases.txt, "
|
||||
"docs/homelab/quadlet/lan.network, "
|
||||
"docs/homelab/scripts/uptime_probe.py, "
|
||||
"docs/homelab/quadlet/compose.container, "
|
||||
"docs/homelab/backups.md, "
|
||||
"docs/homelab/kubernetes.md"
|
||||
), row_log.sources
|
||||
|
||||
Reference in New Issue
Block a user