53 lines
2.2 KiB
Markdown
53 lines
2.2 KiB
Markdown
# Phase 11 — Long Answers (No Truncation)
|
|
|
|
**Story:** `.agent/user_stories/long-answers.md`
|
|
**Context:** owner report 2026-08-22 — "responses keep getting cut off.
|
|
It should be allowed to respond up to 32768 tokens."
|
|
|
|
## Goal
|
|
Remove the hard 700-token output cap on chat answers; the model may
|
|
respond up to **32 768** tokens (`BOR_MAX_OUTPUT_TOKENS`, default
|
|
32 768).
|
|
|
|
## Diagnosis
|
|
`app/rag/llm.py::chat_stream` calls `chat.completions.create(...,
|
|
max_tokens=700, ...)`. Long answers die mid-sentence at ~700 tokens.
|
|
|
|
## Implementation steps
|
|
1. **Config** (`app/config.py`): `max_output_tokens: int = 32_768` in the
|
|
RAG-tuning section (env `BOR_MAX_OUTPUT_TOKENS`); document in
|
|
`.env.example`.
|
|
2. **LLM client** (`app/rag/llm.py`): `chat_stream` passes
|
|
`max_tokens=self.settings.max_output_tokens`.
|
|
3. **E2E mock** (`tests/e2e/mock_llm.py`):
|
|
- Honor `max_tokens` deterministically: token ≈ whitespace word; if
|
|
the composed answer is longer, truncate to the first N words.
|
|
(With the old 700 cap a long answer loses its tail — the mock now
|
|
behaves like the real endpoint.)
|
|
- New trigger: user message containing `write a long answer` →
|
|
deterministic ~4 000-word numbered answer ending in a unique final
|
|
line (`LONG-ANSWER-END`).
|
|
- No behavior change for existing (short) answers: they fit under any
|
|
sane cap.
|
|
4. **Tests:**
|
|
- Unit: settings default + env override (`test_config.py`);
|
|
`chat_stream` forwards the configured `max_tokens` (fake client in
|
|
`test_llm_client.py`).
|
|
- E2E: `tests/e2e/test_long_answers.py` per the story mapping.
|
|
|
|
## Locked decisions
|
|
None touched. A5 (aipi endpoint) unchanged; `turbo` accepts the larger
|
|
cap per owner instruction.
|
|
|
|
## Testing & Quality
|
|
- Unit + integration green; `uv run pytest --cov=app --cov-report=term-missing`
|
|
**>90%**; E2E in isolation:
|
|
`uv run pytest tests/e2e/test_long_answers.py -v --no-cov`.
|
|
- No regressions: `test_chat_rag.py` + `test_chat_api.py` green
|
|
(short answers unaffected by the mock's new `max_tokens` honoring).
|
|
|
|
## Commit
|
|
```bash
|
|
git add -A .agent/ app/ tests/ .env.example && git commit --no-gpg-sign -m "fix(rag): lift chat output cap to 32768 tokens — long answers no longer cut off"
|
|
```
|