phase: 114_embed_question_length
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.
**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
This commit is contained in:
@@ -0,0 +1,195 @@
|
||||
"""Phase 114 E2E (Playwright): the 4,000-char (composer-clamp) question
|
||||
sends a CLEAN turn — the L6 acceptance pin (TODO.md L179–181).
|
||||
|
||||
Source: TODO.md L149–181 — "L6 — 4,000-char question clamp exceeds the
|
||||
embed model's input cap → misleading 'couldn't reach the embedding model'
|
||||
error (2026-09-15, brain-of-reese interactive test)". The repro was 100%
|
||||
reliable: a question at the composer's 4,000-char clamp (~903 tokens)
|
||||
made the REAL aipi endpoint's litellm reject the embedding with
|
||||
``input (903 tokens) is too large to process`` (HTTP 500) — and the
|
||||
turn died pre-token with the banner "I couldn't reach the embedding
|
||||
model — please try again."
|
||||
|
||||
The fix under test (LOCKED A1 + A2, 00_phase.md): the embed step now
|
||||
embeds at most ``embed_question_max_chars`` (default 1200 — the
|
||||
chunker's ``HARD_MAX_CHARS`` budget, env-tunable) of the question —
|
||||
the unit suite (``tests/unit/test_embed_question_length.py``) pins that
|
||||
``embed_one`` receives EXACTLY the 1200-char prefix while the FULL
|
||||
question still reaches the LLM prompt — while the 4,000-char composer
|
||||
clamp stays (locked A2: truncation, not a lower clamp).
|
||||
|
||||
Pinned here (the truncated-embed success path — the unit suite pins the
|
||||
prefix itself and the too-long error mapping):
|
||||
|
||||
* a question typed to the FULL clamp (EXACTLY 4,000 chars — the counter
|
||||
reads ``4000/4000 — character limit`` + ``.is-max``) sends: the turn
|
||||
streams to ``done`` on the mock LLM with the grounded answer marker,
|
||||
NO error banner (the pre-phase "couldn't reach the embedding model"
|
||||
death is gone), the user bubble carries the FULL 4,000-char question,
|
||||
and the input + counter clear (never stale, PLAN §7.4).
|
||||
|
||||
The question repeats a fixture-KB sentence (``kubernetes`` — indexed by
|
||||
``tests/fixtures/docs/homelab/kubernetes.md``) so the hybrid retrieval
|
||||
grounds (the mock-calibrated 0.30 threshold in the e2e conftest) and
|
||||
the brain bubble carries ``MOCK_ANSWER_MARKER`` — the strongest
|
||||
"the turn completed" reading.
|
||||
|
||||
The endpoint and the chat are authed (phase 79, ``require_user``), so
|
||||
the test signs in as admin first (``auth_helpers.login``). The send
|
||||
auto-saves a ``saved_chats`` row, so the autouse fixture truncates that
|
||||
table before and after the test (the phase-80/103 isolation pattern).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from e2e.auth_helpers import login
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
#: The composer's hard cap (``maxlength="4000"`` mirroring the server
|
||||
#: ``ChatRequest.message max_length=4000`` — the phase-104 A3 clamp).
|
||||
_CAP = 4_000
|
||||
|
||||
#: A sentence the fixture KB indexes (``kubernetes`` — the FTS leg of
|
||||
#: the hybrid retrieval hits, so the turn GROUNDS) repeated to the
|
||||
#: clamp: the L6 repro shape — a legal 4,000-char question whose pre-
|
||||
#: phase embedding input (~903 tokens) exceeded the real endpoint's
|
||||
#: per-request input cap.
|
||||
_QUESTION_SENTENCE = (
|
||||
"How is my homelab kubernetes cluster configured for long-running batch jobs? "
|
||||
)
|
||||
QUESTION = (_QUESTION_SENTENCE * 52)[:_CAP]
|
||||
assert len(QUESTION) == _CAP, "the question must land EXACTLY at the clamp"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_chats(db_ready: None) -> Iterator[None]:
|
||||
"""The send auto-saves a row per turn — truncate ``saved_chats``
|
||||
before and after the test so it starts from (and leaves) an empty
|
||||
deployment (the phase-80/103 autouse pattern)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE saved_chats"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE saved_chats"))
|
||||
db.commit()
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
|
||||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return await import_sources([FIXTURES], LLMClient(settings))
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread (Playwright owns the test loop)."""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
t = Thread(target=runner)
|
||||
t.start()
|
||||
t.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _seed_kb(mock_port: int) -> ImportSummary:
|
||||
"""Deterministic KB: truncate the KB tables, import the fixture
|
||||
docs (needed for the grounded answer)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
summary = _run_in_thread(_import_fixtures(mock_port))
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
return summary
|
||||
|
||||
|
||||
def _wait_chat_booted(page: Page) -> None:
|
||||
"""Wait until app.js has FINISHED booting the chat page. The login
|
||||
helper returns on the URL change (navigation commit) — the page's
|
||||
module script may still be executing, and an ``input`` event
|
||||
dispatched before its top-level listener registrations land on a
|
||||
page whose listeners do not exist yet (the event is simply lost).
|
||||
``#view-chat.chat-booted`` is added two frames after the boot
|
||||
settles (AFTER every top-level listener), so it is the "the app's
|
||||
JS is live" sentinel."""
|
||||
page.wait_for_function(
|
||||
"() => document.getElementById('view-chat')?."
|
||||
"classList.contains('chat-booted')",
|
||||
timeout=15_000,
|
||||
)
|
||||
expect(page.locator("#view-chat")).to_have_class(re.compile(r"\bchat-booted\b"))
|
||||
|
||||
|
||||
def test_question_at_the_composer_clamp_sends_a_clean_turn(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""L6 acceptance: a question typed to the FULL 4,000-char clamp
|
||||
(counter ``4000/4000 — character limit`` + ``.is-max``) sends — the
|
||||
embed step sends only the bounded 1200-char prefix to the model
|
||||
(unit-pinned), so the turn streams to ``done`` on the mock LLM:
|
||||
the user bubble carries the FULL 4,000-char question, the brain
|
||||
bubble carries the grounded mock marker, NO error banner (the
|
||||
pre-phase "couldn't reach the embedding model" death), and the
|
||||
input + counter clear."""
|
||||
_seed_kb(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_wait_chat_booted(page)
|
||||
|
||||
counter = page.locator("#char-count")
|
||||
input_el = page.locator("#message-input")
|
||||
banner = page.locator("#kb-banner")
|
||||
expect(counter).to_be_hidden()
|
||||
expect(banner).to_be_hidden()
|
||||
|
||||
# Type the full-clamp question: fill sets the value + dispatches
|
||||
# the input event (the counter path) — EXACTLY 4,000 chars.
|
||||
page.fill("#message-input", QUESTION)
|
||||
expect(input_el).to_have_value(QUESTION)
|
||||
expect(counter).to_be_visible(timeout=5_000)
|
||||
expect(counter).to_have_text("4000/4000 — character limit")
|
||||
expect(counter).to_have_class(re.compile("is-max"))
|
||||
|
||||
# Send at the clamp: the bounded-prefix embed succeeds and the turn
|
||||
# streams to done — no error frame of any kind.
|
||||
page.click("#send-btn")
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(1, timeout=30_000)
|
||||
expect(page.locator(".msg.user .bubble")).to_have_text(QUESTION)
|
||||
brain = page.locator(".msg.brain .bubble").first
|
||||
expect(brain).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
expect(banner).to_be_hidden() # NO "couldn't reach the embedding model" death
|
||||
|
||||
# Never stale: the turn cleared the input AND the counter, and the
|
||||
# send button recovered.
|
||||
expect(input_el).to_have_value("")
|
||||
expect(counter).to_be_hidden()
|
||||
expect(page.locator("#send-btn")).to_be_enabled()
|
||||
Reference in New Issue
Block a user