phase: 108_history_wire_check
Final verification pass complete — everything independently re-verified green. Report: **Phase 108 — History wire check (final verification pass; verdict: NO BUG, tests-only per D13)** - Verified all 3 tasks complete in `.agents/phases/complete/108_history_wire_check/`; re-ran every pin and gate myself - Layer 1 (unit): 2 owner-shape pins in `tests/unit/test_history.py` (2-turn keep-whole + A4 thinking mapping) — pass - Layer 2 (integration): `test_endpoint_two_turn_history_reaches_the_llm` — captured LLM request = `[system, user Q1, assistant R1, user Q2]`, `history_msgs=2` — pass - Layer 3 (E2E): `tests/e2e/test_history_wire_check.py` — owner's verbatim 3-message scenario, echo byte-exact (T2: `2 prior messages`+R1 tail; T3: `4 prior messages`+R2 tail) — pass - `VERDICT.md` present in phase dir: layer outcomes + "no bug — model behavior/user error" + pin names **Gates (exact results)** - `uv run pytest --cov=app --cov-report=term-missing` → 2305 passed, TOTAL **99%** (>90%) - `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` → 2 passed (isolation, DB up) - `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression) → 3 passed - `uv run ruff check .` → clean; `uv run pyright` → 0 errors **Completion criteria:** all six met — no defects found; diff scoped to `tests/**`, `VERDICT.md`, `.agents/phases/**`, `.agents/reports/**` (no `app/`/`frontend/` changes, consistent with no-bug verdict). Per executor rules, no git commit made — left for the harness. **Next pending phase:** `109_turn_progress_loader`
This commit is contained in:
@@ -0,0 +1,231 @@
|
||||
"""Phase 108 (task 02) E2E (Playwright): the owner's exact follow-up
|
||||
scenario, byte-exact on the FULL browser wire.
|
||||
|
||||
Story: n/a — owner bug report 2026-09-16 (TODO.md L4): "I've noticed at
|
||||
least one instance where a follow-up chat is missing the first message
|
||||
and response as context. So if I ask 'What is my name' and then 'What
|
||||
did I just ask you?' the model responds 'This is the first question
|
||||
you've asked'. But if I send a third message 'What was the previous
|
||||
question' the model responds correctly … Just check if there's a bug,
|
||||
there may not be."
|
||||
|
||||
This is LAYER 3 of the phase's three-layer wire check (D13): layers 1-2
|
||||
(the trimmer + the endpoint, task 01) prove the SERVER; this file proves
|
||||
the CLIENT — the localStorage ``bor.chat.v1`` record →
|
||||
``conversation.slice(0, -1)`` mapping → request body → the LLM — with
|
||||
the owner's exact 3-message scenario. The oracle is the phase-74
|
||||
``echo my history`` marker, reused unmodified (D14): the mock answers
|
||||
``history: N prior messages; last answer tail: <last 24 chars of the
|
||||
most recent prior assistant message, or "none">; thinking: yes|no`` —
|
||||
N = non-system messages before the LAST user message (the current
|
||||
question excluded). The marker is checked BEFORE the mock's
|
||||
DEFLECT_MODE branch, so the echo fires on BOTH turn branches — the
|
||||
owner's questions are personal, the gate branch is irrelevant to the
|
||||
pin, and no branch is asserted.
|
||||
|
||||
The expected tails come from the ``bor.chat.v1`` record the client
|
||||
persisted (the SAME array task 02's mapping sends as ``history`` —
|
||||
what the record shows IS what the model received), read AFTER each turn
|
||||
persists, exactly the ``test_llm_history.py`` idiom.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov
|
||||
|
||||
Each test pins:
|
||||
|
||||
* ``test_cold_start_echo_shows_no_phantom_history`` — a FRESH
|
||||
conversation: the first message's echo shows ``0 prior messages`` /
|
||||
``last answer tail: none`` / ``thinking: no`` — no phantom prior
|
||||
turns leak into the first turn's history block.
|
||||
* ``test_owner_scenario_three_turns_carry_the_full_prior_history`` —
|
||||
the owner's exact 3 messages (turns 2-3 carry the echo marker, the
|
||||
owner's words preserved verbatim as the prefix):
|
||||
T1 ``What is my name?`` → R1 (read raw from the record); T2
|
||||
``What did I just ask you? echo my history`` → the echo must show
|
||||
``2 prior messages`` + R1's exact 24-char tail; T3 ``What was the
|
||||
previous question? echo my history`` → ``4 prior messages`` + R2's
|
||||
exact tail. THE regression pin for the owner's symptom: the missing-
|
||||
first-turn bug renders T2's echo as ``0 prior messages`` /
|
||||
``last answer tail: none``.
|
||||
|
||||
No thinking is ever triggered (none of the owner's questions carries
|
||||
the ``think out loud`` marker), so every echo reads
|
||||
``thinking: no``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from e2e.auth_helpers import login
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
STORAGE_KEY = "bor.chat.v1"
|
||||
|
||||
#: The owner's exact words (TODO.md L4, verbatim). Turn 1 stays marker-
|
||||
#: free (its answer R1 is the tail the turn-2 echo must reproduce);
|
||||
#: turns 2-3 append the phase-74 history-echo marker AFTER the owner's
|
||||
#: question, so the marker question is the owner's question plus an
|
||||
#: echo suffix — the prefix travels unchanged in the user message.
|
||||
T1 = "What is my name?"
|
||||
T2 = "What did I just ask you? echo my history"
|
||||
T3 = "What was the previous question? echo my history"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
|
||||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return await import_sources([FIXTURES], LLMClient(settings))
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread.
|
||||
|
||||
Playwright's sync API keeps an asyncio loop running on the test
|
||||
thread, so ``asyncio.run`` cannot be called directly from a test
|
||||
body.
|
||||
"""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
t = Thread(target=runner)
|
||||
t.start()
|
||||
t.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
"""Truncate the KB (+ query log + steering notes — deterministic
|
||||
mock answers), then optionally re-import fixtures."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes"))
|
||||
db.commit()
|
||||
if not seed:
|
||||
return None
|
||||
return _run_in_thread(_import_fixtures(mock_port))
|
||||
|
||||
|
||||
def _ask(page: Page, question: str) -> None:
|
||||
"""Send one turn and wait until the answer has fully landed (the
|
||||
``done`` event restored the Send button)."""
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
MOCK_ANSWER_MARKER, timeout=60_000
|
||||
)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
|
||||
def _record(page: Page) -> dict[str, Any]:
|
||||
"""The persisted ``bor.chat.v1`` record (the same array the client
|
||||
maps into the request body's ``history`` — what it shows IS what
|
||||
the model receives next)."""
|
||||
raw = page.evaluate(f"localStorage.getItem({STORAGE_KEY!r})")
|
||||
return json.loads(raw) if raw else {"messages": []}
|
||||
|
||||
|
||||
def _wait_record(page: Page, n_messages: int) -> dict[str, Any]:
|
||||
"""Wait until the persisted record carries ``n_messages`` turns
|
||||
(the ``done`` event's save point has landed in localStorage)."""
|
||||
page.wait_for_function(
|
||||
"""([key, n]) => {
|
||||
const raw = localStorage.getItem(key);
|
||||
const rec = raw ? JSON.parse(raw) : null;
|
||||
return !!rec && rec.messages.length >= n;
|
||||
}""",
|
||||
arg=[STORAGE_KEY, n_messages],
|
||||
timeout=15_000,
|
||||
)
|
||||
return _record(page)
|
||||
|
||||
|
||||
def test_cold_start_echo_shows_no_phantom_history(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
# Cold start: no restored conversation — the first question's
|
||||
# history block is empty, whatever the mock answers.
|
||||
page.add_init_script("localStorage.clear()")
|
||||
login(page, app_url, next="/")
|
||||
|
||||
# The FIRST message echoes: no prior turns at all — no phantom
|
||||
# question, no phantom answer tail, no phantom thinking.
|
||||
_ask(page, "echo my history")
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("history: 0 prior messages", timeout=30_000)
|
||||
expect(bubble).to_contain_text("last answer tail: none")
|
||||
expect(bubble).to_contain_text("thinking: no")
|
||||
|
||||
|
||||
def test_owner_scenario_three_turns_carry_the_full_prior_history(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
# Cold start: the ONLY prior turns any turn sees are the ones this
|
||||
# test sends — no restored conversation.
|
||||
page.add_init_script("localStorage.clear()")
|
||||
login(page, app_url, next="/")
|
||||
|
||||
# Turn 1 — the owner's exact first question, marker-free. R1's raw
|
||||
# text comes from the persisted record (NOT the rendered DOM): the
|
||||
# record IS the array the client sends as `history` next turn, so
|
||||
# its tail is exactly what the model will receive.
|
||||
_ask(page, T1)
|
||||
r1 = _wait_record(page, 2)["messages"][1]
|
||||
assert r1["who"] == "brain"
|
||||
assert r1["text"], "turn 1 must have persisted a non-empty answer"
|
||||
assert "thinking" not in r1 or not r1["thinking"] # no thinking marker in T1
|
||||
|
||||
# Turn 2 — the owner's exact second question + the echo marker.
|
||||
# THE regression pin: the missing-first-turn bug the owner reported
|
||||
# renders this echo as "history: 0 prior messages" /
|
||||
# "last answer tail: none". A complete wire shows BOTH prior
|
||||
# messages (T1 + R1) and R1's exact 24-char tail.
|
||||
_ask(page, T2)
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("history: 2 prior messages", timeout=30_000)
|
||||
expect(bubble).to_contain_text(f"last answer tail: {r1['text'][-24:]}")
|
||||
expect(bubble).to_contain_text("thinking: no")
|
||||
# The persisted record carries the streamed echo — what the client
|
||||
# saved is what it sends as R2 on turn 3.
|
||||
r2 = _wait_record(page, 4)["messages"][3]
|
||||
assert r2["who"] == "brain"
|
||||
assert "history: 2 prior messages" in r2["text"]
|
||||
|
||||
# Turn 3 — the owner's exact third question + the echo marker: the
|
||||
# FULL prior conversation (T1, R1, T2, R2) must reach the model —
|
||||
# 4 prior messages, the most recent answer being R2 (the echo
|
||||
# itself), tail read from the record again.
|
||||
_ask(page, T3)
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("history: 4 prior messages", timeout=30_000)
|
||||
expect(bubble).to_contain_text(f"last answer tail: {r2['text'][-24:]}")
|
||||
expect(bubble).to_contain_text("thinking: no")
|
||||
r3 = _wait_record(page, 6)["messages"][5]
|
||||
assert r3["who"] == "brain"
|
||||
assert "history: 4 prior messages" in r3["text"]
|
||||
@@ -1733,6 +1733,49 @@ def test_grounded_turn_forwards_history_through_the_agent(
|
||||
assert lines and "history_msgs=4" in lines[-1]
|
||||
|
||||
|
||||
def test_endpoint_two_turn_history_reaches_the_llm(
|
||||
client,
|
||||
db,
|
||||
seeded_kb: FakeRagLLM,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Phase 108 layer 2 (TODO L4 — the owner's follow-up, server wire):
|
||||
the owner's exact 2-turn history (Q1 "What is my name?" / R1 "Your
|
||||
name is Reese.") plus the follow-up "What did I just ask you?" must
|
||||
reach the LLM as ``[system, user Q1, assistant R1, user Q2]`` — the
|
||||
two prior turns NOT dropped (the reported missing-first-turn symptom
|
||||
would be their absence from this captured request). The pin is
|
||||
branch-agnostic BY DESIGN: phase 74 pinned the history splice on
|
||||
BOTH branches, so whichever the seeded KB retrieves to (LOW/deflected
|
||||
or HIGH/grounded), the captured request must carry the full prior
|
||||
exchange. The seeded-KB fixture is used because it is the file's
|
||||
standard idiom (and keeps the cosine gate exercised like production);
|
||||
the canned fake answer ends the turn on exactly one LLM request on
|
||||
either branch."""
|
||||
history = [
|
||||
{"who": "user", "text": "What is my name?"},
|
||||
{"who": "brain", "text": "Your name is Reese."},
|
||||
]
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
frames = _stream_chat_with_history(client, "What did I just ask you?", history)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
assert frames[-1]["type"] == "done" # the turn completes (either branch)
|
||||
assert len(seeded_kb.seen_messages) == 1 # the canned answer ends the turn
|
||||
(messages,) = seeded_kb.seen_messages
|
||||
assert messages[0]["role"] == "system"
|
||||
assert messages[1:-1] == [
|
||||
{"role": "user", "content": "What is my name?"},
|
||||
{"role": "assistant", "content": "Your name is Reese."},
|
||||
] # the FULL prior exchange — chronological, nothing dropped
|
||||
assert messages[-1] == {"role": "user", "content": "What did I just ask you?"}
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "history_msgs=2" in lines[-1]
|
||||
|
||||
|
||||
def test_request_without_history_sends_exactly_system_and_user(
|
||||
client,
|
||||
db,
|
||||
|
||||
@@ -96,6 +96,63 @@ def test_result_is_chronological_oldest_to_newest() -> None:
|
||||
assert "reasoning_content" not in got[3]
|
||||
|
||||
|
||||
# ---------- owner shape (phase 108, TODO L4: the 2-turn follow-up) ----------
|
||||
|
||||
|
||||
def test_short_two_turn_history_kept_whole_and_chronological() -> None:
|
||||
"""Phase 108 layer 1 (TODO L4 — the owner's exact shape): one user
|
||||
question + one brain answer is orders of magnitude under the DEFAULT
|
||||
budgets (40 turns / 24 000 chars — no overrides, the file's plain
|
||||
``_settings()``) and must survive the trimmer WHOLE: both turns kept,
|
||||
chronological (Q1 then R1), roles mapped, no trim, no reordering. A
|
||||
server-side missing-first-turn on the owner's follow-up would have to
|
||||
be dropped here."""
|
||||
history = [
|
||||
_turn("user", "What is my name?"),
|
||||
_turn("brain", "Your name is Reese."),
|
||||
]
|
||||
assert history_to_messages(history, _settings()) == [
|
||||
{"role": "user", "content": "What is my name?"},
|
||||
{"role": "assistant", "content": "Your name is Reese."},
|
||||
]
|
||||
|
||||
|
||||
def test_two_turn_history_thinking_mapping() -> None:
|
||||
"""Phase 108 layer 1 (TODO L4), the A4 gate on the owner's shape: a
|
||||
non-empty prior ``thinking`` travels as ``reasoning_content`` on the
|
||||
assistant message; with ``thinking`` absent or empty the key is
|
||||
ABSENT (not an empty string) — the message equals the plain brain
|
||||
turn."""
|
||||
got = history_to_messages(
|
||||
[
|
||||
_turn("user", "What is my name?"),
|
||||
_turn("brain", "Your name is Reese.", thinking="The owner asked for their name."),
|
||||
],
|
||||
_settings(),
|
||||
)
|
||||
assert got == [
|
||||
{"role": "user", "content": "What is my name?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Your name is Reese.",
|
||||
"reasoning_content": "The owner asked for their name.",
|
||||
},
|
||||
]
|
||||
for thinking in (None, ""): # absent and empty — both omit the key
|
||||
got = history_to_messages(
|
||||
[
|
||||
_turn("user", "What is my name?"),
|
||||
_turn("brain", "Your name is Reese.", thinking=thinking),
|
||||
],
|
||||
_settings(),
|
||||
)
|
||||
assert got == [
|
||||
{"role": "user", "content": "What is my name?"},
|
||||
{"role": "assistant", "content": "Your name is Reese."},
|
||||
]
|
||||
assert "reasoning_content" not in got[1]
|
||||
|
||||
|
||||
# ---------- turn-count budget ----------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user