phase: 114_embed_question_length
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.
**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
This commit is contained in:
@@ -0,0 +1,195 @@
|
||||
"""Phase 114 E2E (Playwright): the 4,000-char (composer-clamp) question
|
||||
sends a CLEAN turn — the L6 acceptance pin (TODO.md L179–181).
|
||||
|
||||
Source: TODO.md L149–181 — "L6 — 4,000-char question clamp exceeds the
|
||||
embed model's input cap → misleading 'couldn't reach the embedding model'
|
||||
error (2026-09-15, brain-of-reese interactive test)". The repro was 100%
|
||||
reliable: a question at the composer's 4,000-char clamp (~903 tokens)
|
||||
made the REAL aipi endpoint's litellm reject the embedding with
|
||||
``input (903 tokens) is too large to process`` (HTTP 500) — and the
|
||||
turn died pre-token with the banner "I couldn't reach the embedding
|
||||
model — please try again."
|
||||
|
||||
The fix under test (LOCKED A1 + A2, 00_phase.md): the embed step now
|
||||
embeds at most ``embed_question_max_chars`` (default 1200 — the
|
||||
chunker's ``HARD_MAX_CHARS`` budget, env-tunable) of the question —
|
||||
the unit suite (``tests/unit/test_embed_question_length.py``) pins that
|
||||
``embed_one`` receives EXACTLY the 1200-char prefix while the FULL
|
||||
question still reaches the LLM prompt — while the 4,000-char composer
|
||||
clamp stays (locked A2: truncation, not a lower clamp).
|
||||
|
||||
Pinned here (the truncated-embed success path — the unit suite pins the
|
||||
prefix itself and the too-long error mapping):
|
||||
|
||||
* a question typed to the FULL clamp (EXACTLY 4,000 chars — the counter
|
||||
reads ``4000/4000 — character limit`` + ``.is-max``) sends: the turn
|
||||
streams to ``done`` on the mock LLM with the grounded answer marker,
|
||||
NO error banner (the pre-phase "couldn't reach the embedding model"
|
||||
death is gone), the user bubble carries the FULL 4,000-char question,
|
||||
and the input + counter clear (never stale, PLAN §7.4).
|
||||
|
||||
The question repeats a fixture-KB sentence (``kubernetes`` — indexed by
|
||||
``tests/fixtures/docs/homelab/kubernetes.md``) so the hybrid retrieval
|
||||
grounds (the mock-calibrated 0.30 threshold in the e2e conftest) and
|
||||
the brain bubble carries ``MOCK_ANSWER_MARKER`` — the strongest
|
||||
"the turn completed" reading.
|
||||
|
||||
The endpoint and the chat are authed (phase 79, ``require_user``), so
|
||||
the test signs in as admin first (``auth_helpers.login``). The send
|
||||
auto-saves a ``saved_chats`` row, so the autouse fixture truncates that
|
||||
table before and after the test (the phase-80/103 isolation pattern).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from e2e.auth_helpers import login
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
#: The composer's hard cap (``maxlength="4000"`` mirroring the server
|
||||
#: ``ChatRequest.message max_length=4000`` — the phase-104 A3 clamp).
|
||||
_CAP = 4_000
|
||||
|
||||
#: A sentence the fixture KB indexes (``kubernetes`` — the FTS leg of
|
||||
#: the hybrid retrieval hits, so the turn GROUNDS) repeated to the
|
||||
#: clamp: the L6 repro shape — a legal 4,000-char question whose pre-
|
||||
#: phase embedding input (~903 tokens) exceeded the real endpoint's
|
||||
#: per-request input cap.
|
||||
_QUESTION_SENTENCE = (
|
||||
"How is my homelab kubernetes cluster configured for long-running batch jobs? "
|
||||
)
|
||||
QUESTION = (_QUESTION_SENTENCE * 52)[:_CAP]
|
||||
assert len(QUESTION) == _CAP, "the question must land EXACTLY at the clamp"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_chats(db_ready: None) -> Iterator[None]:
|
||||
"""The send auto-saves a row per turn — truncate ``saved_chats``
|
||||
before and after the test so it starts from (and leaves) an empty
|
||||
deployment (the phase-80/103 autouse pattern)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE saved_chats"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE saved_chats"))
|
||||
db.commit()
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
|
||||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return await import_sources([FIXTURES], LLMClient(settings))
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread (Playwright owns the test loop)."""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
t = Thread(target=runner)
|
||||
t.start()
|
||||
t.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _seed_kb(mock_port: int) -> ImportSummary:
|
||||
"""Deterministic KB: truncate the KB tables, import the fixture
|
||||
docs (needed for the grounded answer)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
summary = _run_in_thread(_import_fixtures(mock_port))
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
return summary
|
||||
|
||||
|
||||
def _wait_chat_booted(page: Page) -> None:
|
||||
"""Wait until app.js has FINISHED booting the chat page. The login
|
||||
helper returns on the URL change (navigation commit) — the page's
|
||||
module script may still be executing, and an ``input`` event
|
||||
dispatched before its top-level listener registrations land on a
|
||||
page whose listeners do not exist yet (the event is simply lost).
|
||||
``#view-chat.chat-booted`` is added two frames after the boot
|
||||
settles (AFTER every top-level listener), so it is the "the app's
|
||||
JS is live" sentinel."""
|
||||
page.wait_for_function(
|
||||
"() => document.getElementById('view-chat')?."
|
||||
"classList.contains('chat-booted')",
|
||||
timeout=15_000,
|
||||
)
|
||||
expect(page.locator("#view-chat")).to_have_class(re.compile(r"\bchat-booted\b"))
|
||||
|
||||
|
||||
def test_question_at_the_composer_clamp_sends_a_clean_turn(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""L6 acceptance: a question typed to the FULL 4,000-char clamp
|
||||
(counter ``4000/4000 — character limit`` + ``.is-max``) sends — the
|
||||
embed step sends only the bounded 1200-char prefix to the model
|
||||
(unit-pinned), so the turn streams to ``done`` on the mock LLM:
|
||||
the user bubble carries the FULL 4,000-char question, the brain
|
||||
bubble carries the grounded mock marker, NO error banner (the
|
||||
pre-phase "couldn't reach the embedding model" death), and the
|
||||
input + counter clear."""
|
||||
_seed_kb(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_wait_chat_booted(page)
|
||||
|
||||
counter = page.locator("#char-count")
|
||||
input_el = page.locator("#message-input")
|
||||
banner = page.locator("#kb-banner")
|
||||
expect(counter).to_be_hidden()
|
||||
expect(banner).to_be_hidden()
|
||||
|
||||
# Type the full-clamp question: fill sets the value + dispatches
|
||||
# the input event (the counter path) — EXACTLY 4,000 chars.
|
||||
page.fill("#message-input", QUESTION)
|
||||
expect(input_el).to_have_value(QUESTION)
|
||||
expect(counter).to_be_visible(timeout=5_000)
|
||||
expect(counter).to_have_text("4000/4000 — character limit")
|
||||
expect(counter).to_have_class(re.compile("is-max"))
|
||||
|
||||
# Send at the clamp: the bounded-prefix embed succeeds and the turn
|
||||
# streams to done — no error frame of any kind.
|
||||
page.click("#send-btn")
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(1, timeout=30_000)
|
||||
expect(page.locator(".msg.user .bubble")).to_have_text(QUESTION)
|
||||
brain = page.locator(".msg.brain .bubble").first
|
||||
expect(brain).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
expect(banner).to_be_hidden() # NO "couldn't reach the embedding model" death
|
||||
|
||||
# Never stale: the turn cleared the input AND the counter, and the
|
||||
# send button recovered.
|
||||
expect(input_el).to_have_value("")
|
||||
expect(counter).to_be_hidden()
|
||||
expect(page.locator("#send-btn")).to_be_enabled()
|
||||
@@ -668,9 +668,11 @@ def test_chat_mid_stream_failure_yields_error_after_partial_deltas(client, db) -
|
||||
def test_error_event_matches_contract_shape(
|
||||
client, db, seeded_kb, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The SSE error event (PLAN §4) is exactly ``{type, detail}`` — the
|
||||
client's loading-feedback state machine (phase 06) keys off this shape
|
||||
to flip to the error state and re-enable the send button.
|
||||
"""The SSE error event (PLAN §4) is exactly ``{type, detail, hint}`` —
|
||||
the client's loading-feedback state machine (phase 06) keys off the
|
||||
``type``/``detail`` shape to flip to the error state and re-enable
|
||||
the send button; ``hint`` (phase 114, TODO L6) is additive — present
|
||||
as ``null`` on reachability frames, old clients ignore it.
|
||||
``llm_retries=0`` keeps this a single-attempt turn: the contract under
|
||||
test is the error frame itself, not the phase-67 retry loop."""
|
||||
broken = FakeRagLLM(embed_error=EmbeddingError("embeddings endpoint down"))
|
||||
@@ -686,9 +688,10 @@ def test_error_event_matches_contract_shape(
|
||||
|
||||
assert len(frames) == 1
|
||||
event = frames[0]
|
||||
assert set(event.keys()) == {"type", "detail"}
|
||||
assert set(event.keys()) == {"type", "detail", "hint"}
|
||||
assert event["type"] == "error"
|
||||
assert isinstance(event["detail"], str) and event["detail"]
|
||||
assert event["hint"] is None # reachability frame — no too-long hint
|
||||
|
||||
|
||||
def test_chat_db_down_returns_503_json(client, monkeypatch) -> None:
|
||||
@@ -1704,9 +1707,9 @@ def test_deflected_scaffolding_twice_settles_malformed(
|
||||
) -> None:
|
||||
"""(b) The recovery answer is scaffolding again — a second empty
|
||||
reply is terminal: the DEDICATED error frame (the exact copy), no
|
||||
``done``, no query_log row — byte-for-byte today's ``LLMError``
|
||||
terminal shape — and no third request (at most one recovery per
|
||||
turn)."""
|
||||
``done``, no query_log row — the standard ``LLMError`` terminal
|
||||
shape (phase 114: the additive ``hint`` field is ``null`` here) —
|
||||
and no third request (at most one recovery per turn)."""
|
||||
span = _scaffold_span()
|
||||
dead = FakeRagLLM(answer_sequence=[span, span])
|
||||
live = get_settings()
|
||||
@@ -1721,7 +1724,12 @@ def test_deflected_scaffolding_twice_settles_malformed(
|
||||
assert frames[0]["detail"] == (
|
||||
"The model returned a malformed reply — please try again."
|
||||
)
|
||||
assert set(frames[0].keys()) == {"type", "detail"} # the contract shape
|
||||
assert set(frames[0].keys()) == {
|
||||
"type",
|
||||
"detail",
|
||||
"hint",
|
||||
} # the contract shape (phase 114: additive hint — null here)
|
||||
assert frames[0]["hint"] is None
|
||||
assert span not in json.dumps(frames)
|
||||
assert not any(f["type"] == "done" for f in frames)
|
||||
assert db.scalars(select(QueryLog)).all() == []
|
||||
|
||||
@@ -0,0 +1,569 @@
|
||||
"""Unit: the chat question-embed prefix budget (phase 114, TODO L6; LOCKED A1)
|
||||
and the too-large embed error mapping (task 02; LOCKED A3).
|
||||
|
||||
The embed step of ``POST /api/chat`` embeds at most
|
||||
``settings.embed_question_max_chars`` (default 1200 — the chunker's
|
||||
``HARD_MAX_CHARS`` budget: worst-case ~1.4 chars/token, so it stays
|
||||
under the endpoint's ~1024-token per-request input cap) of the
|
||||
question; the FULL question still reaches the LLM prompt. A question
|
||||
at or under the budget embeds byte-identically to the pre-phase path.
|
||||
|
||||
Error mapping (task 02): a single text over the endpoint's input cap
|
||||
is a DETERMINISTIC size failure (``EmbeddingInputTooLargeError``, the
|
||||
real ``_post_embeddings`` → ``_TooLarge`` branch) — the chat endpoint
|
||||
settles it with the accurate "question too long" terminal frame + the
|
||||
reachability-fine hint, ONE attempt, NO retry frame (locked A3). An
|
||||
embed failure without the too-large signature keeps the phase-67
|
||||
reachability path byte-identically (retry frames + the old copy).
|
||||
|
||||
The endpoint-level tests drive ``POST /api/chat`` with the LLM (a
|
||||
recording fake that captures every ``embed_one`` input and the
|
||||
messages of each request, or a real ``LLMClient`` on a canned-failure
|
||||
transport for the error-mapping tests), the retriever, and the DB
|
||||
session all faked (the ``test_chat_gate.py`` wiring), so the whole
|
||||
embed → retrieve → prompt contract runs without a stack.
|
||||
|
||||
Frontend pins (task 02, source-assertion house style): the SSE
|
||||
error frame's optional ``hint`` threads through the stream state
|
||||
machine to ``showErrorBanner`` (shown in place of the default
|
||||
reachability hint); the phase-111 Retry button rides the same
|
||||
turn-error path.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api import chat as chat_api
|
||||
from app.config import Settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document, KbOverview
|
||||
from app.rag.chunker import HARD_MAX_CHARS
|
||||
from app.rag.llm import (
|
||||
EmbeddingError,
|
||||
EmbeddingInputTooLargeError,
|
||||
LLMClient,
|
||||
StreamPiece,
|
||||
)
|
||||
from app.rag.retriever import RetrievedChunk
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
|
||||
ANSWER = "Here is what your notes say about that."
|
||||
|
||||
|
||||
def _question(n: int) -> str:
|
||||
"""A deterministic *n*-char question with a distinct head and tail.
|
||||
|
||||
Repeated, index-marked sentences cut at exactly *n* chars — the
|
||||
4,000-char case is the composer's schema clamp
|
||||
(``ChatRequest.message`` ``max_length=4000``), the L6 repro.
|
||||
"""
|
||||
sentence = "How is my homelab kubernetes cluster configured for long-running batch jobs? "
|
||||
parts: list[str] = []
|
||||
total = 0
|
||||
i = 0
|
||||
while total < n:
|
||||
part = f"[{i}] " + sentence
|
||||
parts.append(part)
|
||||
total += len(part)
|
||||
i += 1
|
||||
return "".join(parts)[:n]
|
||||
|
||||
|
||||
# ---------- the setting (default + validator) ----------
|
||||
|
||||
|
||||
def test_default_budget_matches_the_chunker_hard_cap() -> None:
|
||||
"""LOCKED A1: the default is the chunker's ``HARD_MAX_CHARS`` budget."""
|
||||
assert Settings(_env_file=None).embed_question_max_chars == 1200 # pyright: ignore[reportCallIssue]
|
||||
assert Settings.model_fields["embed_question_max_chars"].default == HARD_MAX_CHARS
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", [0, -1, -1200])
|
||||
def test_budget_rejects_zero_and_negative(bad: int) -> None:
|
||||
"""``0``/negative would embed an empty/absent prefix — a typo that
|
||||
must fail loudly at startup (the ``agent_max_rounds`` pattern)."""
|
||||
with pytest.raises(ValidationError, match="embed_question_max_chars must be > 0"):
|
||||
Settings(_env_file=None, embed_question_max_chars=bad) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("good", [1, 500, 10_000])
|
||||
def test_budget_accepts_positive_values(good: int) -> None:
|
||||
"""A model with a smaller/larger cap is env-tunable, no code change."""
|
||||
settings = Settings(_env_file=None, embed_question_max_chars=good) # pyright: ignore[reportCallIssue]
|
||||
assert settings.embed_question_max_chars == good
|
||||
|
||||
|
||||
# ---------- endpoint-level (fake LLM + fake retriever + fake session) ----------
|
||||
|
||||
|
||||
class _RecordingLLM:
|
||||
"""Records every ``embed_one`` input and the messages of each request.
|
||||
|
||||
Streams a canned answer and never emits tool calls, so a grounded
|
||||
turn through the agent loop ends after the single (tools-offered)
|
||||
request. Mirrors the ``test_chat_gate.py`` fake LLM.
|
||||
"""
|
||||
|
||||
def __init__(self, answer: str = ANSWER) -> None:
|
||||
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
self.embedded: list[str] = []
|
||||
self.answer = answer
|
||||
self.seen: list[list[dict[str, str]]] = []
|
||||
|
||||
async def embed_one(self, text: str) -> list[float]:
|
||||
self.embedded.append(text)
|
||||
return [0.0] * 768
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
scaffolding: ScaffoldingFilter | None = None, # phase 71 pass-through
|
||||
):
|
||||
self.seen.append(messages)
|
||||
for i in range(0, len(self.answer), 12):
|
||||
yield StreamPiece("content", self.answer[i : i + 12])
|
||||
|
||||
|
||||
class _FakeSteeringResult:
|
||||
"""Empty steering-note result (no stored notes in these tests)."""
|
||||
|
||||
def all(self) -> list[Any]:
|
||||
return []
|
||||
|
||||
|
||||
class _FakeSession:
|
||||
"""Stands in for the DB session (the ``test_chat_gate.py`` fake)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.added: list[Any] = []
|
||||
self.commits = 0
|
||||
|
||||
def __enter__(self) -> _FakeSession:
|
||||
return self
|
||||
|
||||
def __exit__(self, *args: Any) -> None:
|
||||
pass
|
||||
|
||||
def add(self, obj: Any) -> None:
|
||||
self.added.append(obj)
|
||||
|
||||
def commit(self) -> None:
|
||||
self.commits += 1
|
||||
|
||||
def scalars(self, _stmt: Any) -> _FakeSteeringResult:
|
||||
return _FakeSteeringResult()
|
||||
|
||||
def get(self, model: Any, pk: Any) -> Any:
|
||||
if model is KbOverview:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _doc(title: str, content: str) -> Document:
|
||||
return Document(
|
||||
id=uuid.uuid4(),
|
||||
source="Homelab",
|
||||
path=f"{title.lower().replace(' ', '-')}.md",
|
||||
full_path="/tmp/doc.md",
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
created_at=datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC),
|
||||
)
|
||||
|
||||
|
||||
def _chunk(doc: Document, score: float) -> RetrievedChunk:
|
||||
return RetrievedChunk(
|
||||
chunk_id=uuid.uuid4(),
|
||||
position=0,
|
||||
content=doc.content[:32],
|
||||
score=score,
|
||||
document=doc,
|
||||
cosine=score,
|
||||
fts_hit=False,
|
||||
is_summary=False,
|
||||
)
|
||||
|
||||
|
||||
def _fake_retriever(chunks: list[RetrievedChunk]) -> Any:
|
||||
def retrieve(_db: Any, _question: str, _vec: list[float]) -> list[RetrievedChunk]:
|
||||
return chunks
|
||||
|
||||
return retrieve
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _admin_signed_in(client: TestClient) -> None:
|
||||
"""``POST /api/chat`` is user-gated — the endpoint-level tests run
|
||||
as the signed-in ADMIN (the ``test_chat_gate.py`` pattern)."""
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def embed_env(monkeypatch: pytest.MonkeyPatch) -> Iterator[tuple[_FakeSession, _RecordingLLM]]:
|
||||
"""``POST /api/chat`` with retriever, session, and LLM all faked.
|
||||
|
||||
The code defaults apply (``embed_question_max_chars=1200``); the
|
||||
gate threshold is pinned low so a 0.9-chunk goes grounded and a
|
||||
0.29-chunk deflects, regardless of any local ``.env``.
|
||||
"""
|
||||
monkeypatch.setattr(chat_api, "db_available", lambda: True)
|
||||
session = _FakeSession()
|
||||
llm = _RecordingLLM()
|
||||
monkeypatch.setattr(chat_api, "SessionLocal", lambda: session)
|
||||
monkeypatch.setitem(fastapi_app.dependency_overrides, chat_api.get_llm, lambda: llm)
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
yield session, llm
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def _ask(client: TestClient, message: str) -> list[dict[str, Any]]:
|
||||
with client.stream("POST", "/api/chat", json={"message": message}) as r:
|
||||
assert r.status_code == 200
|
||||
frames: list[dict[str, Any]] = []
|
||||
buf = ""
|
||||
for part in r.iter_text():
|
||||
buf += part
|
||||
while "\n\n" in buf:
|
||||
frame, buf = buf.split("\n\n", 1)
|
||||
frame = frame.strip()
|
||||
if frame.startswith("data:"):
|
||||
frames.append(json.loads(frame.removeprefix("data:").strip()))
|
||||
assert buf.strip() == ""
|
||||
return frames
|
||||
|
||||
|
||||
def test_long_question_embeds_exactly_the_bounded_prefix(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""L6 repro: the 4,000-char (composer-clamp) question embeds ONLY the
|
||||
1200-char prefix — one embed call, exactly the head, and the turn
|
||||
completes (no error frame)."""
|
||||
_session, llm = embed_env
|
||||
question = _question(4_000)
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(_doc("T", "C"), 0.29)]))
|
||||
|
||||
frames = _ask(client, question)
|
||||
|
||||
# The code default (1200), derived from the field so this never drifts.
|
||||
budget = Settings.model_fields["embed_question_max_chars"].default
|
||||
assert llm.embedded == [question[:budget]]
|
||||
assert llm.embedded[0] != question # it really was cut
|
||||
assert all(f["type"] != "error" for f in frames)
|
||||
assert frames[-1]["type"] == "done"
|
||||
|
||||
|
||||
def test_long_question_full_text_reaches_llm_prompt(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Truncation is the embed step ONLY: the deflected turn's request
|
||||
carries the FULL 4,000-char question as the user message."""
|
||||
_session, llm = embed_env
|
||||
question = _question(4_000)
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(_doc("T", "C"), 0.29)]))
|
||||
|
||||
_frames = _ask(client, question)
|
||||
|
||||
assert len(llm.seen) == 1
|
||||
assert llm.seen[0][-1] == {"role": "user", "content": question}
|
||||
assert len(llm.seen[0][-1]["content"]) == 4_000
|
||||
|
||||
|
||||
def test_grounded_turn_agent_request_carries_full_question(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Same contract on the grounded (agent-loop) branch: the single
|
||||
tools-offered request carries the FULL question."""
|
||||
_session, llm = embed_env
|
||||
question = _question(4_000)
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(_doc("T", "C"), 0.90)]))
|
||||
|
||||
frames = _ask(client, question)
|
||||
|
||||
assert frames[-1]["deflected"] is False
|
||||
assert len(llm.seen) == 1
|
||||
assert llm.seen[0][-1] == {"role": "user", "content": question}
|
||||
assert llm.embedded == [question[:1200]] # the prefix, not the full text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n", [100, 900])
|
||||
def test_short_question_embeds_byte_identically(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
n: int,
|
||||
) -> None:
|
||||
"""A question under the budget embeds the WHOLE question — the
|
||||
pre-phase call, byte for byte (one call, the exact string)."""
|
||||
_session, llm = embed_env
|
||||
question = _question(n)
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(_doc("T", "C"), 0.29)]))
|
||||
|
||||
_frames = _ask(client, question)
|
||||
|
||||
assert llm.embedded == [question]
|
||||
|
||||
|
||||
def test_question_at_exactly_the_budget_embeds_whole(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The budget is an INCLUSIVE cap (``[:budget]``): a question exactly
|
||||
1200 chars long embeds in full — no char lost at the boundary."""
|
||||
_session, llm = embed_env
|
||||
question = _question(1_200)
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(_doc("T", "C"), 0.29)]))
|
||||
|
||||
_frames = _ask(client, question)
|
||||
|
||||
assert llm.embedded == [question]
|
||||
|
||||
|
||||
def test_budget_is_env_tunable_via_settings(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""LOCKED A1: the budget is a setting — a deployment with a smaller-cap
|
||||
model lowers it via ``BOR_EMBED_QUESTION_MAX_CHARS`` (here: 500) and
|
||||
the prefix follows, no code change."""
|
||||
_session, llm = embed_env
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.30,
|
||||
embed_question_max_chars=500,
|
||||
),
|
||||
)
|
||||
question = _question(4_000)
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(_doc("T", "C"), 0.29)]))
|
||||
|
||||
_frames = _ask(client, question)
|
||||
|
||||
assert llm.embedded == [question[:500]]
|
||||
assert llm.seen[0][-1] == {"role": "user", "content": question}
|
||||
|
||||
|
||||
# ---------- error mapping (task 02, LOCKED A3) ----------
|
||||
|
||||
#: The aipi/litellm signature of a too-large input (the L6 repro body,
|
||||
#: truncated the way the client sees it — the client keys off "too
|
||||
#: large" in the body).
|
||||
_TOO_LARGE_BODY = (
|
||||
"input (903 tokens) is too large to process. increase the physical "
|
||||
"batch size (current batch size: 512)"
|
||||
)
|
||||
|
||||
|
||||
class _CannedHttpResponse:
|
||||
"""One canned transport reply (status + text body, no JSON)."""
|
||||
|
||||
def __init__(self, status_code: int, text: str) -> None:
|
||||
self.status_code = status_code
|
||||
self.text = text
|
||||
|
||||
|
||||
class _CannedHttp:
|
||||
"""Stands in for the httpx transport the openai client owns.
|
||||
|
||||
Every POST returns the same canned failure and records the request
|
||||
body — the embed attempt counter.
|
||||
"""
|
||||
|
||||
def __init__(self, status_code: int, text: str) -> None:
|
||||
self.status_code = status_code
|
||||
self.text = text
|
||||
self.posts: list[dict[str, Any]] = []
|
||||
|
||||
async def post(
|
||||
self, url: str, *, json: dict[str, Any], headers: dict[str, str] | None = None
|
||||
) -> _CannedHttpResponse:
|
||||
self.posts.append(json)
|
||||
return _CannedHttpResponse(self.status_code, self.text)
|
||||
|
||||
|
||||
def _canned_embed_llm(
|
||||
settings: Settings, status_code: int, text: str
|
||||
) -> tuple[LLMClient, _CannedHttp]:
|
||||
"""A REAL ``LLMClient`` whose transport is a canned failure.
|
||||
|
||||
``embed_one`` runs the real ``_post_embeddings`` → ``_embed_batch``
|
||||
path — the ``_TooLarge`` branch fires for real. The chat stream is
|
||||
never reached: the embed step settles the turn first.
|
||||
"""
|
||||
llm = LLMClient(settings)
|
||||
http = _CannedHttp(status_code, text)
|
||||
llm._client = SimpleNamespace(_client=http) # pyright: ignore[reportAttributeAccessIssue]
|
||||
return llm, http
|
||||
|
||||
|
||||
def test_single_oversized_text_raises_the_too_large_subclass() -> None:
|
||||
"""The single-text ``_TooLarge`` branch of ``LLMClient._embed_batch``
|
||||
raises ``EmbeddingInputTooLargeError`` — still an
|
||||
``EmbeddingError`` (the importer path is a drop-in) with the
|
||||
byte-identical import-oriented message."""
|
||||
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
llm, http = _canned_embed_llm(settings, 500, _TOO_LARGE_BODY)
|
||||
with pytest.raises(EmbeddingInputTooLargeError, match="token cap") as exc:
|
||||
asyncio.run(llm.embed_one("x" * 3000))
|
||||
assert isinstance(exc.value, EmbeddingError)
|
||||
assert str(exc.value) == (
|
||||
"a single 3000-char chunk exceeded the endpoint's per-request input "
|
||||
"token cap — lower BOR_CHUNK_TARGET_CHARS and re-import"
|
||||
)
|
||||
assert len(http.posts) == 1
|
||||
assert llm.embed_batches == 0
|
||||
|
||||
|
||||
def test_too_large_embed_maps_to_terminal_too_long_frame(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""L6 acceptance: the litellm "too large to process" 500 maps to the
|
||||
ACCURATE terminal error — exactly ONE error frame with the precise
|
||||
detail + the reachability-fine hint, NO retry frame, ONE embed
|
||||
attempt (locked A3: deterministic — never retried), no "couldn't
|
||||
reach" copy."""
|
||||
_session, _recording = embed_env
|
||||
settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.30,
|
||||
llm_retry_delay=0.01, # keep the (unused here) budget cheap
|
||||
)
|
||||
monkeypatch.setattr(chat_api, "get_settings", lambda: settings)
|
||||
llm, http = _canned_embed_llm(settings, 500, _TOO_LARGE_BODY)
|
||||
monkeypatch.setitem(fastapi_app.dependency_overrides, chat_api.get_llm, lambda: llm)
|
||||
|
||||
frames = _ask(client, "How is my homelab kubernetes cluster configured?")
|
||||
|
||||
assert len(http.posts) == 1 # ONE attempt — no restart (locked A3)
|
||||
assert [f["type"] for f in frames] == ["error"] # terminal: no retry, no done
|
||||
frame = frames[0]
|
||||
assert frame["detail"] == "Question too long — trim it and re-ask."
|
||||
assert frame["hint"] == (
|
||||
"The app reached the embedding model fine — only the question length is the problem."
|
||||
)
|
||||
|
||||
|
||||
def test_transport_embed_failure_keeps_the_legacy_retry_path(
|
||||
client: TestClient,
|
||||
embed_env: tuple[_FakeSession, _RecordingLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Regression pin: an embed 500 WITHOUT the too-large signature
|
||||
keeps the phase-67 reachability behavior byte-identical — one
|
||||
retry frame per restart (attempts 2–4 of 4), then the OLD
|
||||
"couldn't reach" copy (``hint`` null) after the full attempt
|
||||
budget."""
|
||||
_session, _recording = embed_env
|
||||
settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.30,
|
||||
llm_retry_delay=0.01, # 3 restarts × 0.01 s — the shape is what is pinned
|
||||
)
|
||||
monkeypatch.setattr(chat_api, "get_settings", lambda: settings)
|
||||
llm, http = _canned_embed_llm(settings, 500, "internal server error")
|
||||
monkeypatch.setitem(fastapi_app.dependency_overrides, chat_api.get_llm, lambda: llm)
|
||||
|
||||
frames = _ask(client, "How is my homelab kubernetes cluster configured?")
|
||||
|
||||
assert len(http.posts) == 4 # the full budget was spent (it retried)
|
||||
assert [f["type"] for f in frames] == ["retry", "retry", "retry", "error"]
|
||||
for i, frame in enumerate(frames[:3], start=2):
|
||||
assert frame == {"type": "retry", "attempt": i, "max_attempts": 4}
|
||||
error = frames[-1]
|
||||
assert error["detail"] == (
|
||||
"I couldn't reach the embedding model — please try again."
|
||||
)
|
||||
assert error["hint"] is None # the additive field is null, never a too-long hint
|
||||
|
||||
|
||||
# ---------- frontend hint support (task 02, source-assertion house style) ----------
|
||||
|
||||
APP_JS = Path(__file__).resolve().parents[2] / "frontend" / "assets" / "app.js"
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return APP_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_sse_error_branch_threads_the_frame_hint() -> None:
|
||||
"""The stream state machine's error branch carries the frame's
|
||||
optional ``hint`` through the throw (``err.hint``) — the banner
|
||||
shows it in place of the default reachability hint."""
|
||||
js = _js()
|
||||
idx = js.find('ev.type === "error"')
|
||||
assert idx != -1, "the readSSE handler must branch on error frames"
|
||||
end = js.find("throw err;", idx)
|
||||
assert end != -1, "the error branch throws the detail as the turn error"
|
||||
branch = js[idx:end]
|
||||
assert "err.hint = ev.hint;" in branch, (
|
||||
"the frame's optional hint must ride the thrown error"
|
||||
)
|
||||
|
||||
|
||||
def test_catch_passes_the_thrown_hint_to_set_ui_state() -> None:
|
||||
"""The turn catch passes the thrown error's hint to ``setUiState``
|
||||
(the third argument) — the banner call gets it via the opts merge.
|
||||
Non-Error throws pass no opts (the default hint applies)."""
|
||||
js = _js()
|
||||
idx = js.find("setUiState(UI_STATE.error, detail,")
|
||||
assert idx != -1, "the turn-error landing must flow through setUiState"
|
||||
call = js[idx : js.find(");", idx)]
|
||||
assert "{ hint: err.hint }" in call, "the thrown hint must reach setUiState"
|
||||
|
||||
|
||||
def test_set_ui_state_merges_opts_into_the_banner_call() -> None:
|
||||
"""``setUiState``'s error transition keeps the phase-111
|
||||
``{ retryable: true }`` (the banner Retry button) AND merges the
|
||||
turn opts (the hint) into the ``showErrorBanner`` call."""
|
||||
js = _js()
|
||||
idx = js.find("export function setUiState")
|
||||
assert idx != -1
|
||||
body = js[idx : js.find("\n}\n", idx)]
|
||||
assert "opts = {}" in body, "the opts parameter carries the turn hint"
|
||||
assert "showErrorBanner(errorDetail, { retryable: true, ...opts });" in body
|
||||
|
||||
|
||||
def test_show_error_banner_honors_opts_hint() -> None:
|
||||
"""``showErrorBanner`` shows ``opts.hint`` in place of the default
|
||||
``ERROR_HINT`` — in BOTH the with-detail and the detail-less forms
|
||||
(``??`` falls back on null/undefined, so hint-less frames keep the
|
||||
old copy byte-identically)."""
|
||||
js = _js()
|
||||
idx = js.find("function showErrorBanner")
|
||||
assert idx != -1
|
||||
body = js[idx : js.find("\n}\n", idx)]
|
||||
assert body.count("opts.hint ?? ERROR_HINT") == 2, (
|
||||
"the hint fallback must cover the detail and no-detail forms"
|
||||
)
|
||||
@@ -56,16 +56,35 @@ def test_multi_line_text_stays_one_frame() -> None:
|
||||
|
||||
def test_error_event_model_serializes_exact_frame() -> None:
|
||||
"""The ``ChatErrorEvent`` model is the wire shape of every server-side
|
||||
failure the UI's state machine (phase 06) must recover from."""
|
||||
failure the UI's state machine (phase 06) must recover from.
|
||||
|
||||
Phase 114: the additive ``hint`` field serializes ``null`` when
|
||||
absent (old clients ignore the field — PLAN §4; the
|
||||
``ChatDoneEvent.related`` pattern)."""
|
||||
frame = sse_event(ChatErrorEvent(detail="boom").model_dump())
|
||||
assert frame == 'data: {"type": "error", "detail": "boom"}\n\n'
|
||||
assert _payload(frame) == {"type": "error", "detail": "boom"}
|
||||
assert frame == 'data: {"type": "error", "detail": "boom", "hint": null}\n\n'
|
||||
assert _payload(frame) == {"type": "error", "detail": "boom", "hint": None}
|
||||
|
||||
|
||||
def test_error_event_shape_is_type_and_detail_only() -> None:
|
||||
def test_error_event_shape_is_type_detail_and_optional_hint() -> None:
|
||||
dumped = ChatErrorEvent(detail="The chat model dropped the connection").model_dump()
|
||||
assert set(dumped.keys()) == {"type", "detail"}
|
||||
assert set(dumped.keys()) == {"type", "detail", "hint"}
|
||||
assert dumped["type"] == "error" # default — call sites never spell it out
|
||||
assert dumped["hint"] is None # absent hint serializes null, not dropped
|
||||
|
||||
|
||||
def test_error_event_hint_serializes_verbatim_when_set() -> None:
|
||||
"""Phase 114 (TODO L6): the too-long frame carries the reachability-
|
||||
fine hint — it survives the roundtrip byte-identically (em-dash and
|
||||
all) for the banner to show in place of the default hint."""
|
||||
hint = "The app reached the embedding model fine — only the question length is the problem."
|
||||
dumped = ChatErrorEvent(
|
||||
detail="Question too long — trim it and re-ask.", hint=hint
|
||||
).model_dump()
|
||||
assert dumped["hint"] == hint
|
||||
payload = _payload(sse_event(dumped))
|
||||
assert payload["hint"] == hint
|
||||
assert payload["detail"] == "Question too long — trim it and re-ask."
|
||||
|
||||
|
||||
def test_thinking_frame_serializes_exactly() -> None:
|
||||
|
||||
Reference in New Issue
Block a user