phase: 108_history_wire_check
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 12s

Final verification pass complete — everything independently re-verified green. Report:

**Phase 108 — History wire check (final verification pass; verdict: NO BUG, tests-only per D13)**

- Verified all 3 tasks complete in `.agents/phases/complete/108_history_wire_check/`; re-ran every pin and gate myself
- Layer 1 (unit): 2 owner-shape pins in `tests/unit/test_history.py` (2-turn keep-whole + A4 thinking mapping) — pass
- Layer 2 (integration): `test_endpoint_two_turn_history_reaches_the_llm` — captured LLM request = `[system, user Q1, assistant R1, user Q2]`, `history_msgs=2` — pass
- Layer 3 (E2E): `tests/e2e/test_history_wire_check.py` — owner's verbatim 3-message scenario, echo byte-exact (T2: `2 prior messages`+R1 tail; T3: `4 prior messages`+R2 tail) — pass
- `VERDICT.md` present in phase dir: layer outcomes + "no bug — model behavior/user error" + pin names

**Gates (exact results)**
- `uv run pytest --cov=app --cov-report=term-missing` → 2305 passed, TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` → 2 passed (isolation, DB up)
- `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression) → 3 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors

**Completion criteria:** all six met — no defects found; diff scoped to `tests/**`, `VERDICT.md`, `.agents/phases/**`, `.agents/reports/**` (no `app/`/`frontend/` changes, consistent with no-bug verdict). Per executor rules, no git commit made — left for the harness.

**Next pending phase:** `109_turn_progress_loader`
This commit is contained in:
2026-09-13 23:57:54 -04:00
parent 6bcee831ec
commit fbbd98d734
20 changed files with 833 additions and 0 deletions
+231
View File
@@ -0,0 +1,231 @@
"""Phase 108 (task 02) E2E (Playwright): the owner's exact follow-up
scenario, byte-exact on the FULL browser wire.
Story: n/a — owner bug report 2026-09-16 (TODO.md L4): "I've noticed at
least one instance where a follow-up chat is missing the first message
and response as context. So if I ask 'What is my name' and then 'What
did I just ask you?' the model responds 'This is the first question
you've asked'. But if I send a third message 'What was the previous
question' the model responds correctly … Just check if there's a bug,
there may not be."
This is LAYER 3 of the phase's three-layer wire check (D13): layers 1-2
(the trimmer + the endpoint, task 01) prove the SERVER; this file proves
the CLIENT — the localStorage ``bor.chat.v1`` record →
``conversation.slice(0, -1)`` mapping → request body → the LLM — with
the owner's exact 3-message scenario. The oracle is the phase-74
``echo my history`` marker, reused unmodified (D14): the mock answers
``history: N prior messages; last answer tail: <last 24 chars of the
most recent prior assistant message, or "none">; thinking: yes|no`` —
N = non-system messages before the LAST user message (the current
question excluded). The marker is checked BEFORE the mock's
DEFLECT_MODE branch, so the echo fires on BOTH turn branches — the
owner's questions are personal, the gate branch is irrelevant to the
pin, and no branch is asserted.
The expected tails come from the ``bor.chat.v1`` record the client
persisted (the SAME array task 02's mapping sends as ``history`` —
what the record shows IS what the model received), read AFTER each turn
persists, exactly the ``test_llm_history.py`` idiom.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov
Each test pins:
* ``test_cold_start_echo_shows_no_phantom_history`` — a FRESH
conversation: the first message's echo shows ``0 prior messages`` /
``last answer tail: none`` / ``thinking: no`` — no phantom prior
turns leak into the first turn's history block.
* ``test_owner_scenario_three_turns_carry_the_full_prior_history`` —
the owner's exact 3 messages (turns 2-3 carry the echo marker, the
owner's words preserved verbatim as the prefix):
T1 ``What is my name?`` → R1 (read raw from the record); T2
``What did I just ask you? echo my history`` → the echo must show
``2 prior messages`` + R1's exact 24-char tail; T3 ``What was the
previous question? echo my history`` → ``4 prior messages`` + R2's
exact tail. THE regression pin for the owner's symptom: the missing-
first-turn bug renders T2's echo as ``0 prior messages`` /
``last answer tail: none``.
No thinking is ever triggered (none of the owner's questions carries
the ``think out loud`` marker), so every echo reads
``thinking: no``.
"""
from __future__ import annotations
import asyncio
import json
from pathlib import Path
from threading import Thread
from typing import Any
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.config import Settings
from app.db import SessionLocal
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
STORAGE_KEY = "bor.chat.v1"
#: The owner's exact words (TODO.md L4, verbatim). Turn 1 stays marker-
#: free (its answer R1 is the tail the turn-2 echo must reproduce);
#: turns 2-3 append the phase-74 history-echo marker AFTER the owner's
#: question, so the marker question is the owner's question plus an
#: echo suffix — the prefix travels unchanged in the user message.
T1 = "What is my name?"
T2 = "What did I just ask you? echo my history"
T3 = "What was the previous question? echo my history"
async def _import_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return await import_sources([FIXTURES], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread.
Playwright's sync API keeps an asyncio loop running on the test
thread, so ``asyncio.run`` cannot be called directly from a test
body.
"""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
"""Truncate the KB (+ query log + steering notes — deterministic
mock answers), then optionally re-import fixtures."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes"))
db.commit()
if not seed:
return None
return _run_in_thread(_import_fixtures(mock_port))
def _ask(page: Page, question: str) -> None:
"""Send one turn and wait until the answer has fully landed (the
``done`` event restored the Send button)."""
page.fill("#message-input", question)
page.click("#send-btn")
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
MOCK_ANSWER_MARKER, timeout=60_000
)
expect(page.locator("#send-label")).to_have_text("Send")
def _record(page: Page) -> dict[str, Any]:
"""The persisted ``bor.chat.v1`` record (the same array the client
maps into the request body's ``history`` — what it shows IS what
the model receives next)."""
raw = page.evaluate(f"localStorage.getItem({STORAGE_KEY!r})")
return json.loads(raw) if raw else {"messages": []}
def _wait_record(page: Page, n_messages: int) -> dict[str, Any]:
"""Wait until the persisted record carries ``n_messages`` turns
(the ``done`` event's save point has landed in localStorage)."""
page.wait_for_function(
"""([key, n]) => {
const raw = localStorage.getItem(key);
const rec = raw ? JSON.parse(raw) : null;
return !!rec && rec.messages.length >= n;
}""",
arg=[STORAGE_KEY, n_messages],
timeout=15_000,
)
return _record(page)
def test_cold_start_echo_shows_no_phantom_history(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
# Cold start: no restored conversation — the first question's
# history block is empty, whatever the mock answers.
page.add_init_script("localStorage.clear()")
login(page, app_url, next="/")
# The FIRST message echoes: no prior turns at all — no phantom
# question, no phantom answer tail, no phantom thinking.
_ask(page, "echo my history")
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text("history: 0 prior messages", timeout=30_000)
expect(bubble).to_contain_text("last answer tail: none")
expect(bubble).to_contain_text("thinking: no")
def test_owner_scenario_three_turns_carry_the_full_prior_history(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
# Cold start: the ONLY prior turns any turn sees are the ones this
# test sends — no restored conversation.
page.add_init_script("localStorage.clear()")
login(page, app_url, next="/")
# Turn 1 — the owner's exact first question, marker-free. R1's raw
# text comes from the persisted record (NOT the rendered DOM): the
# record IS the array the client sends as `history` next turn, so
# its tail is exactly what the model will receive.
_ask(page, T1)
r1 = _wait_record(page, 2)["messages"][1]
assert r1["who"] == "brain"
assert r1["text"], "turn 1 must have persisted a non-empty answer"
assert "thinking" not in r1 or not r1["thinking"] # no thinking marker in T1
# Turn 2 — the owner's exact second question + the echo marker.
# THE regression pin: the missing-first-turn bug the owner reported
# renders this echo as "history: 0 prior messages" /
# "last answer tail: none". A complete wire shows BOTH prior
# messages (T1 + R1) and R1's exact 24-char tail.
_ask(page, T2)
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text("history: 2 prior messages", timeout=30_000)
expect(bubble).to_contain_text(f"last answer tail: {r1['text'][-24:]}")
expect(bubble).to_contain_text("thinking: no")
# The persisted record carries the streamed echo — what the client
# saved is what it sends as R2 on turn 3.
r2 = _wait_record(page, 4)["messages"][3]
assert r2["who"] == "brain"
assert "history: 2 prior messages" in r2["text"]
# Turn 3 — the owner's exact third question + the echo marker: the
# FULL prior conversation (T1, R1, T2, R2) must reach the model —
# 4 prior messages, the most recent answer being R2 (the echo
# itself), tail read from the record again.
_ask(page, T3)
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text("history: 4 prior messages", timeout=30_000)
expect(bubble).to_contain_text(f"last answer tail: {r2['text'][-24:]}")
expect(bubble).to_contain_text("thinking: no")
r3 = _wait_record(page, 6)["messages"][5]
assert r3["who"] == "brain"
assert "history: 4 prior messages" in r3["text"]
+43
View File
@@ -1733,6 +1733,49 @@ def test_grounded_turn_forwards_history_through_the_agent(
assert lines and "history_msgs=4" in lines[-1]
def test_endpoint_two_turn_history_reaches_the_llm(
client,
db,
seeded_kb: FakeRagLLM,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Phase 108 layer 2 (TODO L4 — the owner's follow-up, server wire):
the owner's exact 2-turn history (Q1 "What is my name?" / R1 "Your
name is Reese.") plus the follow-up "What did I just ask you?" must
reach the LLM as ``[system, user Q1, assistant R1, user Q2]`` — the
two prior turns NOT dropped (the reported missing-first-turn symptom
would be their absence from this captured request). The pin is
branch-agnostic BY DESIGN: phase 74 pinned the history splice on
BOTH branches, so whichever the seeded KB retrieves to (LOW/deflected
or HIGH/grounded), the captured request must carry the full prior
exchange. The seeded-KB fixture is used because it is the file's
standard idiom (and keeps the cosine gate exercised like production);
the canned fake answer ends the turn on exactly one LLM request on
either branch."""
history = [
{"who": "user", "text": "What is my name?"},
{"who": "brain", "text": "Your name is Reese."},
]
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
try:
caplog.set_level(logging.INFO, logger="app.chat")
frames = _stream_chat_with_history(client, "What did I just ask you?", history)
finally:
fastapi_app.dependency_overrides.clear()
assert frames[-1]["type"] == "done" # the turn completes (either branch)
assert len(seeded_kb.seen_messages) == 1 # the canned answer ends the turn
(messages,) = seeded_kb.seen_messages
assert messages[0]["role"] == "system"
assert messages[1:-1] == [
{"role": "user", "content": "What is my name?"},
{"role": "assistant", "content": "Your name is Reese."},
] # the FULL prior exchange — chronological, nothing dropped
assert messages[-1] == {"role": "user", "content": "What did I just ask you?"}
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
assert lines and "history_msgs=2" in lines[-1]
def test_request_without_history_sends_exactly_system_and_user(
client,
db,
+57
View File
@@ -96,6 +96,63 @@ def test_result_is_chronological_oldest_to_newest() -> None:
assert "reasoning_content" not in got[3]
# ---------- owner shape (phase 108, TODO L4: the 2-turn follow-up) ----------
def test_short_two_turn_history_kept_whole_and_chronological() -> None:
"""Phase 108 layer 1 (TODO L4 — the owner's exact shape): one user
question + one brain answer is orders of magnitude under the DEFAULT
budgets (40 turns / 24 000 chars — no overrides, the file's plain
``_settings()``) and must survive the trimmer WHOLE: both turns kept,
chronological (Q1 then R1), roles mapped, no trim, no reordering. A
server-side missing-first-turn on the owner's follow-up would have to
be dropped here."""
history = [
_turn("user", "What is my name?"),
_turn("brain", "Your name is Reese."),
]
assert history_to_messages(history, _settings()) == [
{"role": "user", "content": "What is my name?"},
{"role": "assistant", "content": "Your name is Reese."},
]
def test_two_turn_history_thinking_mapping() -> None:
"""Phase 108 layer 1 (TODO L4), the A4 gate on the owner's shape: a
non-empty prior ``thinking`` travels as ``reasoning_content`` on the
assistant message; with ``thinking`` absent or empty the key is
ABSENT (not an empty string) — the message equals the plain brain
turn."""
got = history_to_messages(
[
_turn("user", "What is my name?"),
_turn("brain", "Your name is Reese.", thinking="The owner asked for their name."),
],
_settings(),
)
assert got == [
{"role": "user", "content": "What is my name?"},
{
"role": "assistant",
"content": "Your name is Reese.",
"reasoning_content": "The owner asked for their name.",
},
]
for thinking in (None, ""): # absent and empty — both omit the key
got = history_to_messages(
[
_turn("user", "What is my name?"),
_turn("brain", "Your name is Reese.", thinking=thinking),
],
_settings(),
)
assert got == [
{"role": "user", "content": "What is my name?"},
{"role": "assistant", "content": "Your name is Reese."},
]
assert "reasoning_content" not in got[1]
# ---------- turn-count budget ----------