chore(agent): phase roadmap from TODO.md, 2 phases (108 history-wire check, 109 turn progress loader)
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
# Phase 108 — History wire check: verify (or fix) the "missing first turn" follow-up bug
|
||||
|
||||
**Source:** `TODO.md` L4 (owner 2026-09-16): "I've noticed at least one instance where a follow-up chat is missing the first message and response as context. So if I ask 'What is my name' and then 'What did I just ask you?' the model responds 'This is the first question you've asked'. But if I send a third message 'What was the previous question' the model responds correctly 'What did I just ask you?' Just check if there's a bug, there may not be and this was user error"
|
||||
|
||||
**Story:** n/a (owner bug report, phase-74 follow-up — the phase's E2E proves the wire end to end).
|
||||
|
||||
**Context (traced 2026-09-16):** the chat-history wire landed in phase 74 and is three layers deep: (1) the CLIENT maps the `bor.chat.v1` conversation record minus the current question into the request body — `conversation.slice(0, -1)` → `{who, text, thinking?}` per turn (`frontend/assets/app.js` L2247, invariant comment L2229-2246 — the phase-49 retry and phase-53 stale-regen paths pop the old answer before re-sending, so `slice(0,-1)` is exactly the prior turns); (2) the SERVER trims + maps — `history_to_messages` (`app/rag/prompts.py` L199-252): walks NEWEST-FIRST, keeps turns while BOTH budgets hold (`history_max_turns` default **40**, `history_max_chars` default **24_000** — `app/config.py` L94/L101), drops a whole turn on overflow, returns the kept window chronological; `user`→user, `brain`→assistant with `reasoning_content` ONLY when thinking is non-empty (A4); (3) the ENDPOINT splices the block between the system prompt and the current user message on BOTH turn branches (deflected + grounded — `app/api/chat.py` L353-358, "BOTH branches below … reuse the same block"). A short 2-turn conversation is orders of magnitude under both budgets, and both the client mapping and the trimmer READ correctly — so this phase is a deterministic three-layer VERIFICATION with a built-in fix branch, not a rewrite. The wire oracle already exists: the mock LLM's `HISTORY_TRIGGER = "echo my history"` (`tests/e2e/mock_llm.py` L597; checked at L1671 BEFORE the DEFLECT_MODE branch — the echo fires on both branches) answers with `_history_echo(body)` (L1492): a byte-stable `history: N prior messages; last answer tail: <last 24 chars of the most recent prior assistant message, or "none">; thinking: yes|no` — exactly what the owner's scenario needs. The phase-74 E2E (`tests/e2e/test_llm_history.py`) already asserts on this echo, deriving the expected tail from the localStorage record; the existing suites to extend live at `tests/unit/test_history.py` (the pure trimmer) and `tests/integration/test_chat_api.py` (the `HISTORY`/`HISTORY_MESSAGES` idiom L1620-1656 + the `_stream_chat_with_history` helper L1659).
|
||||
|
||||
## Objective
|
||||
Prove — at the trimmer, the endpoint, and the full browser wire — that a follow-up question carries the COMPLETE prior conversation (the owner's exact 2- and 3-turn scenarios, byte-exact via the history echo), and either ship the minimal fix at the layer that reproduces the missing-first-turn symptom or record the verdict "no bug — model behavior/user error" with the pins as the permanent guard.
|
||||
|
||||
## Dependencies
|
||||
- `74_llm_chat_history` (complete) — the feature under verification: the `history` request field, the trimmer, both-branch splicing, the echo marker, and the suites this phase extends. All its pins are regression gates.
|
||||
- `17_thinking_display` (complete) — the record's `thinking` key and the A4 `reasoning_content` wire convention the echo's `thinking: yes|no` term covers.
|
||||
- `49_retry_answer` / `53_stale_saved_chats` (complete) — the client paths (retry, stale-regen) that POP the old answer before re-sending; the client-invariant comment names them — if the client layer ever reproduces, their pop logic is the first suspect.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
- **The three layers, each isolating a suspect (tasks 01-02):**
|
||||
1. **Unit — the trimmer** (task 01): the owner's exact shape — a 2-turn history (user Q1, brain R1) under the default budgets → ALL turns kept, chronological, roles mapped, thinking mapped (non-empty → `reasoning_content`, empty/absent → key absent). If this fails, the bug is in `history_to_messages` and nothing else needs running.
|
||||
2. **Integration — the endpoint** (task 01): the SAME 2-turn history through the real `POST /api/chat` (the `test_chat_api.py::_stream_chat_with_history` idiom): the SSE turn completes AND the LLM request the turn made carries exactly `[system, user Q1, assistant R1, user Q2]` (captured per the house fake-LLM pattern). If layer 1 passes and this fails, the bug is in the endpoint plumbing (the `request.history` → `hist` → prompt splice, one of the two branches).
|
||||
3. **E2E — the full client wire** (task 02): the owner's exact 3-message scenario in the browser, echo markers on turns 2 and 3 (see task 02 for the messages + expected echoes). If layers 1-2 pass and this fails, the bug is in the CLIENT record→history mapping (push/pop timing, the phase-49/53 paths, localStorage restore).
|
||||
- **The verdict (task 03, D13):** all three green → NO BUG: the wire is proven complete at every layer; the reported instance is model behavior/user error (the owner's own hypothesis). The pins stay as the permanent guard (a future regression that drops the first turn fails layer 1, 2, or 3). A failure at layer N → the bug reproduces at layer N; the executor makes the MINIMAL fix in that layer, re-runs the failing layer green, and the verdict records the fix + evidence. `VERDICT.md` (NEW file inside this phase dir) is written BEFORE the commit and states: the layer outcomes, the verdict, and (if fixed) the one-line root cause.
|
||||
- **NOT touched (D13/D14):** the A10 stateless contract, the budget defaults (40 turns / 24k chars), the `bor.chat.v1` record schema, the echo's format (the existing marker IS the oracle — D14: NO new mock marker this phase), any phase-74 pin (regression), `PLAN.md`, completed phases.
|
||||
|
||||
## Tasks
|
||||
1. `01_server_wire_verification.md` — layer 1 (unit pins on the trimmer for the owner's 2-turn shape) + layer 2 (integration: real endpoint, captured LLM request = full prior history).
|
||||
2. `02_client_e2e_owner_scenario.md` — layer 3: new E2E `tests/e2e/test_history_wire_check.py` (isolation) — the owner's exact 3-message scenario, byte-exact echo assertions on turns 2 and 3.
|
||||
3. `03_verdict_fix_or_pin.md` — read the layer outcomes; fix the reproducing layer (or record "no bug"); `VERDICT.md`; full gate; atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit — `tests/unit/test_history.py` (extended): the 2-turn-under-budget keep-all pin + the role/thinking mapping for that shape (the existing budget/trim pins stay green — regression).
|
||||
- Integration — `tests/integration/test_chat_api.py` (extended): the 2-turn request through the real endpoint with the captured-LLM-request assertion (the house fake-LLM capture pattern; the existing phase-74 history tests stay green).
|
||||
- E2E (mandatory, A16) — `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` with the DB up.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Layer 1 green: a 2-turn history under the default budgets survives `history_to_messages` whole, chronological, correctly mapped (unit pin).
|
||||
- [ ] Layer 2 green: a real `POST /api/chat` with a 2-turn history makes the LLM request `[system, user Q1, assistant R1, user Q2]` — the server wire is proven complete (or the bug is fixed here).
|
||||
- [ ] Layer 3 green: the owner's scenario in the browser — turn 2's echo shows `history: 2 prior messages` + R1's exact 24-char tail; turn 3's echo shows `history: 4 prior messages` + R2's tail (or the bug is fixed at the client).
|
||||
- [ ] `VERDICT.md` exists in the phase dir: layer outcomes + the verdict (fixed-at-layer-N with root cause, or "no bug — model behavior") — written before the commit.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` green in isolation; `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase 74) green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit (message per the verdict — see Commit); phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **D13 — Verify-or-fix protocol (owner-instructed: "Just check if there's a bug, there may not be").** The phase's deliverable is the three-layer pins + a recorded verdict. Code changes happen ONLY when a layer reproduces the missing-turn symptom, are MINIMAL, and are confined to the reproducing layer — no A10 contract change, no budget-default change, no record-schema change, no new endpoint. If no layer reproduces, the phase ships tests-only.
|
||||
- **D14 — The existing echo IS the oracle.** The phase-74 `echo my history` marker (`_history_echo`) is reused unmodified — its `N prior messages` count + `last answer tail` are exactly the owner-scenario assertions; NO new mock marker is added this phase (new markers land only in phases that change prompt/tool shapes).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
# verdict = no bug (tests-only):
|
||||
git add tests/ .agents/phases/ && git commit --no-gpg-sign -m "test(chat): history-wire verification pins — TODO L4 verdict: no bug (model behavior)"
|
||||
|
||||
# verdict = bug found (adjust <layer> to the fix site):
|
||||
git add <fixed files> tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): <layer> — follow-up turns carry the full prior history (TODO L4)"
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 01 — Server wire verification: the trimmer (unit) + the endpoint (integration)
|
||||
|
||||
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — "a follow-up chat is missing the first message and response as context … Just check if there's a bug."
|
||||
|
||||
## Objective
|
||||
Prove (or disprove) the two SERVER layers of the history wire for the owner's exact 2-turn shape: a short history must survive `history_to_messages` whole and reach the LLM as the complete prior conversation on the real `POST /api/chat`.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_history.py` (EXISTING — extend, keep every pin green) — add the owner-shape pins for the DEFAULT budgets (no env overrides; construct `Settings` the file's existing `_settings()` way):
|
||||
- `test_short_two_turn_history_kept_whole_and_chronological` — `history = [user "What is my name?", brain "Your name is Reese."]` (the owner's own Q1/R1) → `history_to_messages` returns exactly `[{"role": "user", "content": "What is my name?"}, {"role": "assistant", "content": "Your name is Reese."}]` — both turns, chronological, no trim, no reordering.
|
||||
- `test_two_turn_history_thinking_mapping` — the same 2-turn history with the brain turn carrying a non-empty `thinking` → the assistant message gains `reasoning_content` (A4); with `thinking` empty/absent → the key is ABSENT (not an empty string).
|
||||
- (If either pin fails: STOP — layer 1 reproduces the bug. Fix `app/rag/prompts.py::history_to_messages` minimally (D13), keep this task's pins + the existing suite green, and note the root cause for task 03's `VERDICT.md`. Do not touch the budget defaults.)
|
||||
2. `tests/integration/test_chat_api.py` (EXISTING — extend next to the phase-74 history block, L1620-1670) — add the endpoint-layer pin:
|
||||
- Reuse the file's `_stream_chat`/`_stream_chat_with_history` helpers + fake-LLM capture pattern (read the file's existing setup first — match its house idiom for capturing what the LLM was called with).
|
||||
- `test_endpoint_two_turn_history_reaches_the_llm` — `POST /api/chat {message: "What did I just ask you?", history: [{who: user, text: "What is my name?"}, {who: brain, text: "Your name is Reese."}]}` → the SSE stream completes (`done`), and the chat request the turn made to the LLM carries, IN ORDER, the system prompt, `user "What is my name?"`, `assistant "Your name is Reese."`, then the current `user` question — i.e. the 2 prior turns are NOT dropped (the owner's symptom would be their absence). Assert on the captured `messages` list (roles + contents, exact).
|
||||
- The turn may be LOW/deflected with an empty KB (the history block is branch-independent — pinned in phase 74) or HIGH with one seeded fixture doc (the file's existing seeding idiom) — either is fine; pick what the file's helpers make easiest and say so in a comment.
|
||||
- (If this fails while layer 1 passed: the bug is in the endpoint plumbing — `app/api/chat.py`'s `request.history` → `hist` → prompt splice. Fix minimally (D13), keep this pin + the phase-74 pins green, note the root cause for task 03.)
|
||||
3. Run `uv run pytest tests/unit/test_history.py tests/integration/test_chat_api.py -v` (DB up: `podman compose up -d db`) — green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the owner-shape trimmer pins (keep-all + mapping) alongside the existing budget pins.
|
||||
- Integration: the real endpoint with a captured LLM request — the server wire proven (or fixed) at the exact layer.
|
||||
- Coverage: **>90%** on `app/` (no `app/` change unless a fix is needed; the gate still passes).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `tests/unit/test_history.py` green with the two new owner-shape pins (or the trimmer fixed + pinned)
|
||||
- [ ] `tests/integration/test_chat_api.py` green with `test_endpoint_two_turn_history_reaches_the_llm` (or the endpoint fixed + pinned)
|
||||
- [ ] Every pre-existing pin in both files still green (no regression)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean; the layer-1/layer-2 outcome is noteable for task 03's `VERDICT.md` (pass, or pass-after-fix with root cause)
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 02 — Client E2E: the owner's exact 3-message scenario, byte-exact via the history echo
|
||||
|
||||
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — the owner's repro: Q1 "What is my name?" → Q2 "What did I just ask you?" (model claims it's the first question) → Q3 "What was the previous question?" (model answers correctly).
|
||||
|
||||
## Objective
|
||||
Prove (or disprove) the THIRD layer — the full browser wire: the localStorage conversation record → `conversation.slice(0,-1)` mapping → request body → the LLM — using the owner's exact scenario and the phase-74 `echo my history` oracle (D14: no new marker).
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_history_wire_check.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_llm_history.py`: module-scoped mock-LLM app, the fixture-docs import for a non-empty KB, `e2e.auth_helpers.login`, the localStorage `bor.chat.v1` record reads, per-test conversation reset; module docstring: story n/a — owner bug report 2026-09-16, the isolation command, and what each test pins). Isolation: `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` (DB up).
|
||||
- **The echo oracle, recalled** (mock `_history_echo`, byte-stable): `history: N prior messages; last answer tail: <LAST 24 CHARS of the most recent prior assistant message's content, or "none">; thinking: yes|no` — N = non-system messages before the LAST user message (the current question excluded); checked BEFORE the DEFLECT_MODE branch, so the echo fires whatever gate branch the turn takes (the owner's questions may deflect — that's fine, the marker is in the USER message).
|
||||
- **Tests:**
|
||||
1. `test_cold_start_echo_shows_no_phantom_history` — fresh conversation; ask `echo my history` as the FIRST message → the answer bubble contains `history: 0 prior messages; last answer tail: none; thinking: no` (the cold-start pin: no phantom prior turns).
|
||||
2. `test_owner_scenario_three_turns_carry_the_full_prior_history` — the owner's exact scenario, echo marker APPENDED to turns 2 and 3 (their words preserved verbatim as the prefix):
|
||||
- T1: `What is my name?` → R1 (the mock's deterministic answer — read R1's raw text from the `bor.chat.v1` record's brain entry, NOT from the rendered DOM).
|
||||
- T2: `What did I just ask you? echo my history` → R2's bubble text must contain `history: 2 prior messages; last answer tail: {R1[-24:]}; thinking: no` (R1 = the record's brain text; `thinking: no` — T1 never triggered the thinking marker). **THE regression pin: the owner's bug renders this as `0 prior messages` / `last answer tail: none`.**
|
||||
- T3: `What was the previous question? echo my history` → R3's bubble text must contain `history: 4 prior messages; last answer tail: {R2[-24:]}` (R2 = the echo answer itself — also from the record).
|
||||
- Read the expected tails from the localStorage record AFTER each turn persists (the `test_llm_history.py` pattern — the record the client saved IS what the client sends next, so what the record shows is what the model received).
|
||||
- If this test fails while tasks 01's layers passed: the bug is in the CLIENT mapping (suspects, in order: the `conversation.slice(0,-1)` sites, the phase-49 retry / phase-53 stale-regen pop paths, the record persistence timing — `frontend/assets/app.js` L2187/L2247). Fix minimally (D13), keep this test + `tests/e2e/test_llm_history.py` green, note the root cause for task 03's `VERDICT.md`.
|
||||
2. Run the suite in isolation — green (or pass-after-fix).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E (the task IS the test): the full browser wire, byte-exact via the existing echo oracle.
|
||||
- Coverage: `--no-cov` suite; it exercises `app/` (chat endpoint, history mapping) for real — the `app/` >90% gate is unaffected.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` green in isolation (DB up) — both tests
|
||||
- [ ] The regression pin holds: turn 2's echo shows `2 prior messages` + R1's exact tail; turn 3 shows `4 prior messages` + R2's exact tail (or the client bug is fixed + pinned)
|
||||
- [ ] `tests/e2e/test_llm_history.py` (phase 74) still green in isolation (regression)
|
||||
- [ ] The layer-3 outcome is noteable for task 03's `VERDICT.md` (pass, or pass-after-fix with root cause)
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 03 — Verdict (fix or pin), full gate, atomic commit
|
||||
|
||||
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — "Just check if there's a bug, there may not be and this was user error"; AGENTS.md rules 8/9 — the test gates are non-negotiable.
|
||||
|
||||
## Objective
|
||||
Record the phase's verdict with its evidence, run the complete quality gate, and land the single `--no-gpg-sign` commit — tests-only if no bug was found (the owner's expected outcome), fix + tests if one layer reproduced.
|
||||
|
||||
## Work
|
||||
1. **The verdict** — read the layer outcomes from tasks 01-02 (their test results + any fix notes):
|
||||
- **All three layers green (no fix needed):** the wire is proven complete at the trimmer, the endpoint, and the full browser wire → verdict **NO BUG**: the owner's reported instance was model behavior/user error. The pins stay as the permanent guard (a future regression that drops the first turn fails layer 1, 2, or 3).
|
||||
- **A layer reproduced (pass-after-fix):** the bug is fixed at that layer → verdict **BUG FOUND + FIXED at <layer>**, with the one-line root cause.
|
||||
- Write `.agents/phases/todo/108_history_wire_check/VERDICT.md` BEFORE the commit: the three layer outcomes (pass / pass-after-fix + root cause / fail-should-not-occur), the verdict, and the evidence (which test names carry the pins). Keep it short — it is the durable record the owner asked for ("just check").
|
||||
2. **The full gate** (DB up: `podman compose up -d db`; every command must pass before the commit):
|
||||
- `uv run pytest` — green.
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90%.
|
||||
- E2E in isolation: `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` (NEW) and `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression).
|
||||
- `uv run ruff check . && uv run pyright` — clean.
|
||||
- No-regression spot check: `git diff --stat` shows ONLY the files this phase may touch — `tests/**`, `VERDICT.md`, `.agents/phases/**`, and (only if a bug was fixed) the single reproducing layer's file. If the diff shows anything else, stop and fix the scope before committing.
|
||||
3. **The commit** (exactly one, `--no-gpg-sign`, per the 00_phase.md branch):
|
||||
- no bug: `git add tests/ .agents/phases/ && git commit --no-gpg-sign -m "test(chat): history-wire verification pins — TODO L4 verdict: no bug (model behavior)"`
|
||||
- bug found: `git add <fixed files> tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): <layer> — follow-up turns carry the full prior history (TODO L4)"`
|
||||
4. Move the phase directory: `mv .agents/phases/todo/108_history_wire_check .agents/phases/complete/` (the pipeline gate does this on success — do it only after the commit, and match how prior phases recorded the move).
|
||||
|
||||
## Testing & Quality
|
||||
- This task runs, not writes, the gate: every command above must pass before the commit exists.
|
||||
- Coverage: **>90%** on `app/` (TOTAL line of the `term-missing` report).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `VERDICT.md` in the phase dir: layer outcomes + verdict (no bug / fixed-at-<layer> + root cause) + the pin test names
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
|
||||
- [ ] `tests/e2e/test_history_wire_check.py` + `tests/e2e/test_llm_history.py` green in isolation (DB up)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean; the diff is scoped to this phase's allowed files
|
||||
- [ ] Exactly one new commit with the verdict-branch message, `--no-gpg-sign`; `git status` clean afterwards (only gitignored runtime artifacts aside)
|
||||
- [ ] Phase dir at `.agents/phases/complete/108_history_wire_check/`
|
||||
Reference in New Issue
Block a user