chore(agent): phase roadmap from TODO.md (phases 73-75), clear the file
Convert the two unchecked TODO items into executable phases (Protocol B, appended after the 72 completed phases): - 73_hidden_tab_stream (TODO L3): a merely-hidden browser tab must never stop a generating answer; repro/root-cause decision tree + the pagehide partial-correlation fix + the hidden-tab E2E pin. - 74_llm_chat_history (TODO L4, history): client-provided history in POST /api/chat (stateless, A10) mapped through both the deflected and grounded agent paths, prior thinking blocks preserved via reasoning_content, capped oldest-first; mock echo marker + E2E. - 75_save_doc_full_session (TODO L4, save-as-doc): the Save-as-doc draft body becomes the full session transcript; edit-out happens in the existing doc-edit body; multi-turn git-verified E2E. Owner-confirmed assumptions A1-A7 are recorded as ASSUMPTION lines in the task files. TODO.md is cleared (items now live in .agents/phases/todo/).
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# Phase 74 — Chat history (with prior thinking) reaches the LLM
|
||||
|
||||
**Source:** `TODO.md` L4 — "Chat history isn't being passed to the LLM. When the LLM responds and you ask a follow-up question the previous question/answer isn't passed to the model. Since my models support preserve thinking, make sure to pass previous thinking blocks as well. Then, update the \"save as doc\" process to include the output from the entire chat session rather than the last response. The user can edit out anything they don't want to keep from previous replies." (this phase covers the first three sentences; the "save as doc" sentences are phase 75)
|
||||
**Story:** n/a (TODO-derived)
|
||||
**Context:** `app/api/chat.py` (today `messages = [system, user]` only — the deflected path at ~L381 and the grounded `run_agent` call at ~L414), `app/rag/agent.py` (`run_agent` builds its own `[system, user]` at ~L863), `app/rag/llm.py` (`chat_stream`/`chat`/`chat_stream_retried` take `list[dict[str, str]]`; the wire already carries prior-model reasoning as `reasoning_content` deltas), `app/schemas.py` (`ChatRequest`), `app/config.py` (`BOR_` settings), `frontend/assets/app.js` (`runTurn` ~L1859 is the SINGLE funnel for send / retry-reask / stale-regen; the `conversation` array is the `bor.chat.v1` record whose brain entries already carry the `thinking` key since phase 17), `tests/e2e/mock_llm.py` (the deterministic marker-trigger convention).
|
||||
|
||||
## Objective
|
||||
A follow-up question reaches the model together with the conversation so far — every prior user/brain turn, and the prior thinking blocks on brain turns (preserve-thinking, via the endpoint's existing `reasoning_content` convention) — while `/api/chat` stays stateless: the client provides the history in the request body and the server stores nothing new (owner-locked A10).
|
||||
|
||||
## Dependencies
|
||||
- `73_hidden_tab_stream` (todo) — keeps the chat turn path green first (its E2E pins the stream/record lifecycle this phase's client change touches).
|
||||
|
||||
## Tasks
|
||||
1. `01_server_history_ingestion.md` — `ChatRequest.history` schema + caps settings + the history→messages mapper (assistant turns carry `reasoning_content`) + wiring through BOTH the deflected path and `run_agent` + the per-turn log line + unit/integration tests.
|
||||
2. `02_client_sends_history.md` — `runTurn` sends the `conversation` record (minus the current question, with `thinking`) as `history`; retry/reask and stale-regen ride the same single funnel.
|
||||
3. `03_mock_marker_e2e.md` — the mock's `echo my history` marker + the story E2E proving prior Q/A **and** prior thinking blocks reach the model on grounded, deflected, and first-question paths; regressions + commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: mapper (mapping, thinking passthrough, oldest-first trim on turn AND char budgets, empty default), schema validation (bad `who`, oversized list/text), message ordering in BOTH branches (grounded `run_agent` + deflected), the per-turn log line's new field — in `tests/unit/` + `tests/integration/test_chat_api.py`.
|
||||
- Coverage: **>90%** on `app/` (all new server code is unit/integration tested).
|
||||
- E2E: new story suite `tests/e2e/test_llm_history.py`, run in isolation, against the deterministic mock (its new `echo my history` marker makes the wire contents byte-assertable).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `POST /api/chat` with a `history` body: both the deflected turn and the grounded agent turn send the prior turns to the model in chronological order, with `reasoning_content` on prior brain turns that had thinking (proven by the mock-echo E2E).
|
||||
- [ ] A request without `history` is byte-identical in behavior to today (default empty — existing suites' mocks see no extra messages; regression runs green).
|
||||
- [ ] The per-turn log line (PLAN §9) gains `history_msgs=N`; `uv run pytest` green; coverage >90%; ruff + pyright clean.
|
||||
- [ ] One atomic `--no-gpg-sign` commit (e.g. `feat(rag): pass chat history with prior thinking to the LLM`); phase dir moved to `.agents/phases/complete/`.
|
||||
@@ -0,0 +1,48 @@
|
||||
# Task 01 — Server: optional client-provided history, trimmed + mapped (incl. `reasoning_content`)
|
||||
|
||||
**Phase:** `74_llm_chat_history` · **Source:** `TODO.md:4` — "Chat history isn't being passed to the LLM. When the LLM responds and you ask a follow-up question the previous question/answer isn't passed to the model. Since my models support preserve thinking, make sure to pass previous thinking blocks as well."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`POST /api/chat` accepts an optional `history` (the client's prior turns) and feeds it to the model on BOTH turn branches — the deflected path and the grounded agent — with prior thinking blocks preserved via the endpoint's existing `reasoning_content` wire convention; oversized histories are trimmed oldest-first by configurable caps.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — two new `Settings` fields (existing docstring style, `BOR_`-overridable):
|
||||
- `history_max_turns: int = 40` (`BOR_HISTORY_MAX_TURNS`) — newest history turns kept.
|
||||
- `history_max_chars: int = 24000` (`BOR_HISTORY_MAX_CHARS`) — total kept history chars (text + thinking).
|
||||
2. `app/schemas.py` — next to `ChatRequest`:
|
||||
- `class HistoryTurn(BaseModel)`: `who: Literal["user", "brain"]`, `text: str = Field(min_length=1, max_length=4000)`, `thinking: str | None = Field(default=None, max_length=32000)`.
|
||||
- `ChatRequest.history: list[HistoryTurn] = Field(default_factory=list, max_length=100)` — schema-level hard ceiling (DoS sanity); the config budgets below do the real trimming.
|
||||
3. `app/rag/prompts.py` — new pure helper (unit-testable, no I/O):
|
||||
```python
|
||||
def history_to_messages(history: Sequence[HistoryTurn], settings: Settings) -> list[dict[str, Any]]
|
||||
```
|
||||
- Maps newest-first until BOTH budgets are hit (turn count ≤ `history_max_turns`; cumulative chars — `len(text) + len(thinking or "")` — ≤ `history_max_chars`), then returns the kept turns in chronological (oldest→newest) order.
|
||||
- A budget-exceeding turn is DROPPED WHOLE — never cut mid-answer.
|
||||
- `who="user"` → `{"role": "user", "content": text}`.
|
||||
- `who="brain"` → `{"role": "assistant", "content": text}` plus `"reasoning_content": thinking` ONLY when `thinking` is non-empty (the preserve-thinking wire convention `app/rag/llm.py` already reads on the response side — `delta.reasoning_content`).
|
||||
4. `app/api/chat.py`:
|
||||
- After `settings = get_settings()` in `stream()`: `hist = history_to_messages(request.history, settings)` (once per turn; the deflected recovery below reuses it via `messages`).
|
||||
- Deflected branch: build `messages = [{"role": "system", "content": plan.system_prompt}, *hist, {"role": "user", "content": request.message}]` (replaces the two-entry list at ~L381). The phase-71 recovery already rebuilds from `messages[1:]` — unchanged.
|
||||
- Grounded branch: pass `history=hist` into the `run_agent(...)` call (~L414).
|
||||
- Per-turn log line (PLAN §9, the `logger.info` at ~L636): add `history_msgs={len(hist)}` to the existing line (append it near the other context-size fields, e.g. after `kb_chars`).
|
||||
5. `app/rag/agent.py` — `run_agent(..., history: Sequence[dict[str, Any]] = (), ...)`: build `messages = [{"role": "system", "content": system_prompt}, *history, {"role": "user", "content": user_message}]` (~L863). Everything downstream (tool rounds, phase-71 recovery rebuilding from `messages`, retry restarts) already operates on `messages` — unchanged.
|
||||
6. `app/rag/llm.py` — widen the message parameter type of `chat`, `chat_stream`, and `chat_stream_retried` from `list[dict[str, str]]` to `list[dict[str, Any]]` (an assistant message may now carry `reasoning_content`; the openai SDK passes message dicts through to the request body verbatim, so no transport change — string-only messages stay byte-identical on the wire).
|
||||
7. Tests:
|
||||
- `tests/unit/` (extend the existing prompts/chat test files, e.g. `tests/unit/test_chat_gate.py` or a new `tests/unit/test_history.py`): `history_to_messages` — empty default → `[]`; user/brain mapping; `reasoning_content` present only when thinking non-empty; turn-cap trim (newest kept, oldest dropped whole); char-cap trim (text+thinking accounted, drop-whole semantics); chronological order of the result.
|
||||
- `tests/unit/test_config.py`: the two new settings + `BOR_` env overrides (existing pattern).
|
||||
- `tests/unit/test_agent.py`: `run_agent` with a non-empty `history` places it between system and the current user message (mock `LLMClient` capturing the sent messages); default `history=()` keeps the two-message request.
|
||||
- `tests/integration/test_chat_api.py`: `POST /api/chat` with `history` — deflected turn (off-topic message) and grounded turn (on-topic message) both forward the prior turns to the mock LLM client (assert on the captured request messages, including `reasoning_content`); a request without `history` sends exactly `[system, user]` (byte-identical behavior pin); schema rejection: `who="alien"` → 422; `history` > 100 entries → 422; the per-turn log line carries `history_msgs=N` (caplog).
|
||||
- - ASSUMPTION A2 (owner-confirmed 2026-09-08): `/api/chat` stays STATELESS per owner-locked A10 — the history is client-provided in the request body; the server stores nothing new (no `saved_chats`/`query_log` change beyond the log field).
|
||||
- - ASSUMPTION A3 (owner-confirmed 2026-09-08): history is sent for BOTH grounded and deflected turns; caps = newest 40 turns / 24 000 chars (new `BOR_HISTORY_MAX_TURNS` / `BOR_HISTORY_MAX_CHARS`); a capped-out turn is dropped whole, never truncated.
|
||||
- - ASSUMPTION A4 (owner-confirmed 2026-09-08): prior thinking travels as `reasoning_content` on the assistant message (the convention `app/rag/llm.py` already documents for the response side), only when non-empty — this is what makes the owner's preserve-thinking models keep the reasoning chain.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration as listed in Work item 7 — every branch of the mapper and both API branches is covered.
|
||||
- Coverage: **>90%** on `app/` for this task's new/modified code (the mapper is pure — easy to pin).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `history_to_messages` unit tests green (mapping, `reasoning_content` gating, both budgets, drop-whole, ordering).
|
||||
- [ ] Integration: deflected AND grounded turns forward history to the mock LLM; no-`history` requests are unchanged; 422 pins; log line gains `history_msgs=N`.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No commit in this task (task 03 commits the phase) — but `uv run .agents/validate.sh`-equivalent gates must pass.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 02 — Client: `runTurn` sends the conversation history (with thinking) in the request body
|
||||
|
||||
**Phase:** `74_llm_chat_history` · **Source:** `TODO.md:4` — "Chat history isn't being passed to the LLM. When the LLM responds and you ask a follow-up question the previous question/answer isn't passed to the model. Since my models support preserve thinking, make sure to pass previous thinking blocks as well."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The browser sends the prior conversation on every turn — user turns, brain answers, and the brain turns' `thinking` blocks — as the new `history` field of the `POST /api/chat` body, so follow-up questions are answered WITH the conversation the user has already seen.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — in `runTurn` (~L1859), the single funnel for ALL turns (fresh send, phase-49 retry via `runTurn(text, { reask: true })`, phase-53 stale-regen — verify the invariant while editing: in every one of those paths the question being sent is the LAST entry of `conversation` at fetch time — a fresh send just pushed it; `retryLastTurn` ~L1798 pops the old brain record and keeps the question as the last entry; `regenerateStaleChat` ~L1617 follows the same pop-then-reask pattern):
|
||||
```js
|
||||
const history = conversation.slice(0, -1).map((m) => ({
|
||||
who: m.who,
|
||||
text: m.text,
|
||||
thinking: m.who === "brain" ? m.thinking || undefined : undefined,
|
||||
}));
|
||||
```
|
||||
and send `body: JSON.stringify({ message: text, history })` in the existing `fetch("/api/chat", …)` (~L1905, replacing `{ message: text }`).
|
||||
- `thinking` is only present on brain records that actually streamed one (phase-17+ optional key) — `undefined` drops the key from the JSON (the record's existing convention).
|
||||
- Old/restored records without `thinking` send none — the server maps that to a plain assistant message.
|
||||
- The current question itself is NEVER in `history` (the `slice(0, -1)`) — it is the request's `message`, exactly as the server expects.
|
||||
- No UI change: nothing renders differently; the request body just carries the record the user already sees.
|
||||
2. - ASSUMPTION A5 (owner-confirmed 2026-09-08): stopped/partial brain turns (the `stopped: true` records) are included in the history like any other prior answer — the persisted record is the source of truth, and what the user saw is what the model should treat as the prior answer.
|
||||
3. Sanity check (no code): the server-side schema (task 01) accepts exactly `{who, text, thinking}` — no other record keys (`sources`, `tools`, `deflected`, `suggestions`) travel in the body; `tools` metadata is display-only and was never part of the LLM wire.
|
||||
|
||||
## Testing & Quality
|
||||
- No JS unit-test infra in this repo — the wire-level pin is task 03's E2E (the mock sees the exact messages the client sent).
|
||||
- Manual smoke while running the dev server: DevTools → Network → the second `/api/chat` request's body carries the first Q/A pair (and `thinking` when the first turn streamed one).
|
||||
- `uv run pytest` stays green (frontend-only change).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Every `/api/chat` request carries `history` = the `conversation` record minus the current question, with `thinking` on brain turns that have it (verified in DevTools for a 2-turn conversation).
|
||||
- [ ] Retry (phase 49) and stale-regen (phase 53) send the correct history for their re-asked question (the popped/replaced answer is NOT in the history; the question is NOT duplicated).
|
||||
- [ ] No UI change; `uv run pytest` green; ruff + pyright clean.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 03 — Mock `echo my history` marker + story E2E + regressions + commit
|
||||
|
||||
**Phase:** `74_llm_chat_history` · **Source:** `TODO.md:4` — "Chat history isn't being passed to the LLM. When the LLM responds and you ask a follow-up question the previous question/answer isn't passed to the model. Since my models support preserve thinking, make sure to pass previous thinking blocks as well."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
Prove on the wire, deterministically, that a follow-up question reaches the model WITH the prior Q/A and the prior thinking blocks — grounded and deflected branches alike — then run the regressions and commit the phase.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — new marker, following the module's established convention (trigger phrase in the user message → deterministic, byte-stable answer; checked BEFORE the `DEFLECT_MODE` branch exactly like `TABLE_TRIGGER`, so a marker question always gets the echo whatever the honesty gate says):
|
||||
- `HISTORY_TRIGGER = "echo my history"` — first verify no existing E2E question or fixture file contains the phrase (the module's standing convention).
|
||||
- The echo answer (byte-stable), derived statelessly from the request messages:
|
||||
- `history: N prior messages` — N = the count of non-`system` messages before the LAST `user` message (i.e. everything the client sent as prior turns; the current question itself is excluded).
|
||||
- `last answer tail: <…>` — the LAST 24 chars of the content of the most recent `assistant` message before the current user message, or `none` when there is no prior assistant message.
|
||||
- `thinking: yes|no` — `yes` iff that most recent prior `assistant` message carries a non-empty `reasoning_content` field (the client's phase-74 history mapping), `no` otherwise (or when there is no prior assistant message).
|
||||
- Suffix `(Deterministic mock answer for E2E.)` like the other composed answers.
|
||||
- Docstring entry documenting the marker + the invariant it relies on: **client history contains only `user`/`assistant` messages — never `tool`-role ones** (the client never sends tool calls/results) — so every existing marker flow (which classifies statelessly from tool results) is unaffected by the now-always-present history.
|
||||
2. `tests/e2e/test_llm_history.py` (Playwright; conftest app-boot + KB fixture pattern from `tests/e2e/test_chat_rag.py`; the file name deliberately differs from phase 50's `test_chat_history.py`):
|
||||
- `test_followup_receives_history_and_thinking` — turn 1: an on-topic question carrying `think out loud` (deterministic reasoning stream) → `done`; capture the rendered answer text. Turn 2: an on-topic question carrying `echo my history` → `done`; assert the bubble contains `history: 2 prior messages`, `thinking: yes`, and `last answer tail: ` + the last 24 chars of turn 1's answer (byte-exact from the mock).
|
||||
- `test_first_question_has_no_history` — fresh context, one on-topic `echo my history` question → `history: 0 prior messages`, `thinking: no`, `last answer tail: none` (no phantom history on a cold start).
|
||||
- `test_deflected_followup_receives_history` — turn 1 on-topic with `think out loud`; turn 2 an OFF-TOPIC question carrying `echo my history` (mock embeddings → LOW gate → deflected branch) → the echo answer still appears with `history: 2 prior messages` + `thinking: yes` (ASSUMPTION A3: both branches carry history — the marker is checked before the `DEFLECT_MODE` branch, so the deflected path is what this test proves).
|
||||
3. Regression runs (isolation, AGENTS.md rule 9) — the client now ALWAYS sends history, so the suites that touch the chat wire run: `test_chat_rag.py`, `test_chat_history.py` (phase 50), `test_agent_document_tools.py`, `test_harness_aligned_tools.py` (the agent's message list changed shape), `test_stop_generation.py`, `test_retry_answer.py` (retry's history = the re-ask, per task 02), `test_response_to_docs.py` (saveAsDoc untouched in this phase).
|
||||
4. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
5. Commit (Conventional Commits, `--no-gpg-sign`) — e.g. `feat(rag): pass chat history with prior thinking to the LLM`; move `.agents/phases/todo/74_llm_chat_history/` → `.agents/phases/complete/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` green in isolation (DB up, mock LLM).
|
||||
- Coverage: **>90%** on `app/` (task 01's new code is unit/integration-pinned; this task adds no `app/` code).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The three E2E tests pass: prior Q/A AND prior `reasoning_content` reach the model on follow-ups (grounded + deflected); cold start carries no history.
|
||||
- [ ] The mock's docstring carries the marker + the user/assistant-only invariant; no existing suite's trigger phrases or fixtures collide with `echo my history`.
|
||||
- [ ] Regression suites green in isolation; full suite + coverage >90% + ruff + pyright clean.
|
||||
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
|
||||
Reference in New Issue
Block a user