fix(rag): lift chat output cap to 32768 tokens — long answers no longer cut off

2026-08-22 11:30:19 -04:00
parent 6ec6181c7b
commit 0da5275eeb
9 changed files with 295 additions and 5 deletions
@@ -0,0 +1,52 @@
+# Phase 11 — Long Answers (No Truncation)
+
+**Story:** `.agent/user_stories/long-answers.md`
+**Context:** owner report 2026-08-22 — "responses keep getting cut off.
+It should be allowed to respond up to 32768 tokens."
+
+## Goal
+Remove the hard 700-token output cap on chat answers; the model may
+respond up to **32 768** tokens (`BOR_MAX_OUTPUT_TOKENS`, default
+32 768).
+
+## Diagnosis
+`app/rag/llm.py::chat_stream` calls `chat.completions.create(...,
+max_tokens=700, ...)`. Long answers die mid-sentence at ~700 tokens.
+
+## Implementation steps
+1. **Config** (`app/config.py`): `max_output_tokens: int = 32_768` in the
+   RAG-tuning section (env `BOR_MAX_OUTPUT_TOKENS`); document in
+   `.env.example`.
+2. **LLM client** (`app/rag/llm.py`): `chat_stream` passes
+   `max_tokens=self.settings.max_output_tokens`.
+3. **E2E mock** (`tests/e2e/mock_llm.py`):
+   - Honor `max_tokens` deterministically: token ≈ whitespace word; if
+     the composed answer is longer, truncate to the first N words.
+     (With the old 700 cap a long answer loses its tail — the mock now
+     behaves like the real endpoint.)
+   - New trigger: user message containing `write a long answer` →
+     deterministic ~4 000-word numbered answer ending in a unique final
+     line (`LONG-ANSWER-END`).
+   - No behavior change for existing (short) answers: they fit under any
+     sane cap.
+4. **Tests:**
+   - Unit: settings default + env override (`test_config.py`);
+     `chat_stream` forwards the configured `max_tokens` (fake client in
+     `test_llm_client.py`).
+   - E2E: `tests/e2e/test_long_answers.py` per the story mapping.
+
+## Locked decisions
+None touched. A5 (aipi endpoint) unchanged; `turbo` accepts the larger
+cap per owner instruction.
+
+## Testing & Quality
+- Unit + integration green; `uv run pytest --cov=app --cov-report=term-missing`
+  **>90%**; E2E in isolation:
+  `uv run pytest tests/e2e/test_long_answers.py -v --no-cov`.
+- No regressions: `test_chat_rag.py` + `test_chat_api.py` green
+  (short answers unaffected by the mock's new `max_tokens` honoring).
+
+## Commit
+```bash
+git add -A .agent/ app/ tests/ .env.example && git commit --no-gpg-sign -m "fix(rag): lift chat output cap to 32768 tokens — long answers no longer cut off"
+```