phase: 164_chat_error_honesty

All gates green. Report: - **Verified (phase 164, all 4 tasks already in `complete/`)**: `HistoryTurn`/`ChatMessage`/`ChatThinkingSegment` caps → 600 000, `history`/saved-messages lists → 500, new `SavedChatCreate/Update` total-8 000 000-char validator; exported `errorDetailText` at the two previously unguarded chat paths (pre-stream `!res.ok` + SSE `error` frame); `FAILED_REQUEST_TEXT` + `bubbleText` threading + phase-114 banner hint; dedicated E2E `tests/e2e/test_chat_error_honesty.py`. - **Final-pass additions**: `tests/integration/test_chat_api.py::test_history_rejects_one_char_over_the_text_cap` (endpoint-level >600k → 422 with the pydantic list-of-objects `msg` — the criterion's upper half was only unit-pinned); README env rows for `BOR_APP_VERSION` + `BOR_SESSION_COOKIE`; restored `test_document_viewer.py`'s tracked screenshot artifact dirtied by the E2E run (via read-only `git show`, no index/history touched). - **Tests/lint (exact commands)**: `uv run pytest --cov=app --cov-report=term-missing` → **3837 passed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_chat_error_honesty.py -v --no-cov` → **4 passed in isolation**; per-file loop over **all 148** E2E files (`uv run pytest tests/e2e/<f> -q --no-cov`) → **148 × exit 0**, incl. `test_failed_turn_retry`, `test_chat_save_payload_limits`, `test_edit_and_resend`, `test_llm_history`, `test_smoke`; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - **Completion criteria**: pre-stream 422 shows the distinct line + verbatim `msg`, `[object Object]` unrepresentable + no `input` in DOM — **met** (unit totality/source pins + E2E asserts 1–2); 525k history accepted (200 settles) / >600k → 422 / saved chat 201 + 8M bound — **met**; transport + SSE-frame settles keep `FAILED_TURN_TEXT`, stop/abort untouched, §7.4 re-enables — **met**; budget trim, `message` 4000, the 7 guarded detail-readers and all SSE shapes byte-identical (diff touches only caps + the two chat paths) — **met**; README/AGENTS full review + proxy deploy note — **met**; commit/`complete/` move = pipeline gate. - **Docs**: README reviewed in full (env tables cross-checked programmatically: 74 `BOR_*` rows vs `Settings` defaults — no mismatches; every `.env.example` var documented; referenced scripts/pages/endpoints/`query_log` columns exist) — found & fixed the 2 undocumented real knobs above; the phase's payload-limits section, failed-turn copy and `client_max_body_size ≥ 16 MB` deploy note verified accurate. `AGENTS.md` verified consistent (rules, gitignore list, quick-reference commands, referenced paths all correct) — no changes needed. - **Deviations**: `ChatThinkingSegment.text` raised 32 000 → 600 000 too (not itemized in A1, but its own docstring mirrors `HistoryTurn.thinking`; without it a long-thinking turn 422s its own auto-save — same mirror invariant). No live-infra changes made (the proxy limit lives in the separate deployment repo; README note only). - **Next pending phase**: none — `164_chat_error_honesty` was the only entry in `todo/`.
2026-10-07 18:00:12 -04:00
parent 6ba46e735a
commit 34d60efbc1
42 changed files with 2601 additions and 84 deletions
@@ -7,6 +7,20 @@ past it raises a pydantic ``ValidationError`` naming the offending field
 Plus the regression pin: a realistic ``bor.chat.v1`` payload validates
 cleanly and round-trips ``model_dump()`` (the stored-shape contract —
 the caps added value bounds only, no key/shape change).
+
+Phase 164 (task 01, locked A1) re-cuts the long-answer chain —
+``HistoryTurn.text``/``.thinking``, ``ChatMessage.text``/``.thinking``
+and the thinking-segment cap to 600 000 (a FULL-cap answer at
+``BOR_MAX_OUTPUT_TOKENS=131_072``, ≈ 524 300 chars), the history list
+to 500 and the saved-messages list to 500, and adds the ONE new bound:
+the ``SavedChatCreate``/``SavedChatUpdate`` total-character budget
+(8 000 000) that keeps the anonymous SEC-05 surface bounded now that a
+single message may hold a full-length answer. The chat-path caps are
+pinned here too (``HistoryTurn`` + ``ChatRequest.history``) — same
+boundaries as the saved-chat mirror, and the 422 ``detail`` SHAPE a cap
+rejection produces is pinned as the server-side half of the phase-164
+serializer contract (a LIST OF OBJECTS — the live ``[object Object]``
+report).
 """
 from __future__ import annotations

@@ -14,11 +28,13 @@ import pytest
 from pydantic import ValidationError

 from app.schemas import (
+    SAVED_CHAT_TOTAL_CHARS_BUDGET,
    ChatMessage,
    ChatRequest,
    ChatSegment,
    ChatThinkingSegment,
    ChatToolSegment,
+    HistoryTurn,
    SavedChatCreate,
    SavedChatUpdate,
    SourceRef,
@@ -27,7 +43,14 @@ from app.schemas import (

 # --- boundary constants (phase 83, A1 — mirror sources in app/schemas.py) ---

-TEXT_CAP = 32_000  # mirrors HistoryTurn.text / .thinking
+#: Phase 164 (locked A1): 32 000 → 600 000. ONE constant for the whole
+#: mirror — ``HistoryTurn.text``/``.thinking`` (the chat path),
+#: ``ChatMessage.text``/``.thinking`` and the ``segments`` thinking
+#: entry: the identical-boundary test IS the proof the phase-83 mirror
+#: invariant survived the raise (raise one side and these pins fail).
+#: 600 000 ≈ a full-cap answer at ``BOR_MAX_OUTPUT_TOKENS=131_072``
+#: (≈ 524 300 chars at ~4 chars/token) with margin.
+TEXT_CAP = 600_000
 SOURCES_CAP = 20
 SUGGESTIONS_CAP = 50
 CHIP_CAP = 200
@@ -37,8 +60,15 @@ PATH_CAP = 1000  # documents.path String(1000)
 TITLE_CAP = 500  # documents.title String(500)
 NAME_CAP = 100  # ToolCall.name
 ARGUMENT_CAP = 2000  # ToolCall.argument
-MESSAGES_CAP = 200  # SavedChatCreate/Update.messages
+MESSAGES_CAP = 500  # SavedChatCreate/Update.messages (phase 164: was 200)
+HISTORY_CAP = 500  # ChatRequest.history (phase 164: was 100)
 QUESTION_CAP = 4_000  # ChatRequest.message — the cap the composer mirrors (phase 104)
+#: The phase-164 anonymous-surface bound, imported from the source (not
+#: restated) so the schema and the suite cannot drift apart.
+TOTAL_BUDGET = SAVED_CHAT_TOTAL_CHARS_BUDGET
+#: A realistic full-cap answer length (131 072 output tokens at ~4
+#: chars/token) — the payload the whole chain must now carry.
+FULL_CAP_ANSWER = 524_300


 def _source_ref() -> dict:
@@ -207,6 +237,103 @@ def test_chat_request_message_one_over_cap_rejects() -> None:
    _failed_loc(exc.value, "message")


+# ---------------------------------------------------------------------------
+# HistoryTurn + ChatRequest.history — the CHAT path of the long-answer
+# chain (phase 164, task 01, locked A1). The live 422 this phase fixes:
+# a >32k-char brain answer rode back as ``history`` on the NEXT turn and
+# rejected it, so the follow-up of any long answer died with
+# "[object Object]". Same boundaries as the saved-chat mirror above.
+# ---------------------------------------------------------------------------
+
+
+def test_history_turn_text_at_cap_validates() -> None:
+    """600 000 chars — a full-cap answer (≈ 524 300 at
+    ``BOR_MAX_OUTPUT_TOKENS=131_072``) flows back as ``history``:
+    the exact request shape the browser sends on the turn AFTER a long
+    answer."""
+    turn = HistoryTurn(who="brain", text="a" * TEXT_CAP)
+    assert len(turn.text) == TEXT_CAP
+    # And the realistic full-cap answer, not just the boundary value.
+    assert len(HistoryTurn(who="brain", text="a" * FULL_CAP_ANSWER).text) == FULL_CAP_ANSWER
+
+
+def test_history_turn_text_one_over_cap_rejects() -> None:
+    with pytest.raises(ValidationError) as exc:
+        HistoryTurn(who="brain", text="a" * (TEXT_CAP + 1))
+    _failed_loc(exc.value, "text")
+
+
+def test_history_turn_thinking_at_cap_validates() -> None:
+    turn = HistoryTurn(who="brain", text="ok", thinking="t" * TEXT_CAP)
+    assert len(turn.thinking or "") == TEXT_CAP
+
+
+def test_history_turn_thinking_one_over_cap_rejects() -> None:
+    with pytest.raises(ValidationError) as exc:
+        HistoryTurn(who="brain", text="ok", thinking="t" * (TEXT_CAP + 1))
+    _failed_loc(exc.value, "thinking")
+
+
+def test_history_turn_caps_match_the_saved_chat_mirror() -> None:
+    """The phase-83 mirror invariant, verified by identical boundaries
+    (not by comment): a payload at the cap validates on BOTH models and
+    one past it fails on BOTH — ``HistoryTurn`` (the chat path) and
+    ``ChatMessage`` (the saved-chat path). Raise one side in
+    ``app/schemas.py`` and this pin fails."""
+    for length, rejected in ((TEXT_CAP, False), (TEXT_CAP + 1, True)):
+        payload = {"who": "brain", "text": "a" * length, "thinking": "t" * length}
+        for model in (HistoryTurn, ChatMessage):
+            if rejected:
+                with pytest.raises(ValidationError):
+                    model.model_validate(payload)
+            else:
+                model.model_validate(payload)
+
+
+def test_chat_request_history_at_cap_validates() -> None:
+    """500 turns pass — the client sends the WHOLE ``bor.chat.v1``
+    record minus the current question, so the ceiling is a DoS sanity
+    bound only (the config budgets trim for real, locked A4)."""
+    req = ChatRequest.model_validate(
+        {"message": "hi", "history": [{"who": "user", "text": f"q{i}"} for i in range(HISTORY_CAP)]}
+    )
+    assert len(req.history) == HISTORY_CAP
+
+
+def test_chat_request_history_one_over_cap_rejects() -> None:
+    with pytest.raises(ValidationError) as exc:
+        ChatRequest.model_validate(
+            {
+                "message": "hi",
+                "history": [
+                    {"who": "user", "text": f"q{i}"} for i in range(HISTORY_CAP + 1)
+                ],
+            }
+        )
+    _failed_loc(exc.value, "history")
+
+
+def test_cap_rejection_detail_is_a_list_of_objects() -> None:
+    """The server-side half of the phase-164 contract (task 02's input):
+    a cap rejection is the pydantic 422 ``detail`` — a LIST OF OBJECTS
+    (``loc``/``msg``/``type``), never a plain string. This is exactly the
+    shape the client turned into ``[object Object]``; the serializer must
+    read ``msg`` and never the ``input`` (which carries the whole
+    conversation)."""
+    with pytest.raises(ValidationError) as exc:
+        HistoryTurn(who="brain", text="a" * (TEXT_CAP + 1))
+    detail = exc.value.errors()
+    assert isinstance(detail, list) and len(detail) == 1
+    (entry,) = detail
+    assert entry["loc"] == ("text",)
+    assert entry["type"] == "string_too_long"
+    # The message the UI shows verbatim in the failed-turn note.
+    assert entry["msg"] == f"String should have at most {TEXT_CAP} characters"
+    # The raw input DOES ride the error object server-side — which is why
+    # the client serializer only ever reads ``msg``.
+    assert entry["input"].startswith("aaaa")
+
+
 # ---------------------------------------------------------------------------
 # SourceRef (documents column-length mirrors)
 # ---------------------------------------------------------------------------
@@ -322,6 +449,97 @@ def test_saved_chat_update_messages_one_over_cap_rejects() -> None:
    _failed_loc(exc.value, "messages")


+# ---------------------------------------------------------------------------
+# The phase-164 total-character budget (locked A1) — the ONE new bound on
+# the anonymous saved-chat surface. The per-message caps went to 600 000,
+# so the LIST is bounded by total characters instead of by an item count
+# that a full-cap answer would blow through: 8 000 000 keeps the worst
+# case at the order the phase-83 SEC-05 audit accepted (200 × 64k
+# ≈ 12.8 MB) while fitting ~15 full-cap answers.
+# ---------------------------------------------------------------------------
+
+
+def _full_cap_messages(count: int, *, size: int = FULL_CAP_ANSWER) -> list[dict]:
+    """``count`` brain answers at the realistic full-cap length."""
+    return [{"who": "brain", "text": "a" * size} for _ in range(count)]
+
+
+def test_saved_chat_within_total_budget_of_full_cap_answers_validates() -> None:
+    """~15 full-cap answers (15 × 524 300 = 7 864 500 chars, the biggest
+    whole-count fit under 8 000 000) validate — the budget is a sanity
+    bound, not a practical limit on a real conversation."""
+    fits = TOTAL_BUDGET // FULL_CAP_ANSWER
+    assert fits == 15
+    payload = SavedChatCreate.model_validate({"messages": _full_cap_messages(fits)})
+    assert len(payload.messages) == fits
+
+
+def test_saved_chat_over_total_budget_rejects_with_the_budget_message() -> None:
+    """One full-cap answer past the fit → ValidationError whose ``msg``
+    NAMES the budget (the 422 the anonymous caller sees), on BOTH the
+    create and the update model — the same validator on both bodies."""
+    over = _full_cap_messages(TOTAL_BUDGET // FULL_CAP_ANSWER + 1)
+    for model in (SavedChatCreate, SavedChatUpdate):
+        with pytest.raises(ValidationError) as exc:
+            model.model_validate({"messages": over})
+        (error,) = exc.value.errors()
+        assert error["loc"] == ()  # a model-level validator, not a field
+        assert "8,000,000-character budget" in error["msg"]
+
+
+def test_saved_chat_one_full_cap_answer_plus_normal_turns_validates() -> None:
+    """The realistic case the live report blocked: a saved chat carrying
+    ONE 525k-char brain answer alongside ordinary turns."""
+    payload = SavedChatCreate.model_validate(
+        {
+            "messages": [
+                {"who": "user", "text": "Write me the whole runbook."},
+                {"who": "brain", "text": "a" * 525_000},
+                {"who": "user", "text": "And the rollback?"},
+                {"who": "brain", "text": "Short rollback."},
+            ]
+        }
+    )
+    assert len(payload.messages) == 4
+
+
+def test_saved_chat_total_budget_counts_thinking() -> None:
+    """``thinking`` counts against the budget exactly like ``text`` (the
+    history budget's own accounting, mirrored): 15 turns of 280 000 text
+    + 280 000 thinking = 8 400 000 REJECTS even though the text alone is
+    only 4 200 000 — a text-only accounting would have waved it
+    through."""
+    turns = 15
+    size = 280_000
+    assert turns * size <= TOTAL_BUDGET < turns * (size * 2)
+    heavy = [
+        {"who": "brain", "text": "a" * size, "thinking": "t" * size}
+        for _ in range(turns)
+    ]
+    with pytest.raises(ValidationError) as exc:
+        SavedChatCreate.model_validate({"messages": heavy})
+    assert "8,000,000-character budget" in exc.value.errors()[0]["msg"]
+    # One turn fewer fits (14 × 560 000 = 7 840 000).
+    assert len(SavedChatCreate.model_validate({"messages": heavy[:-1]}).messages) == 14
+
+
+def test_saved_chat_at_exactly_the_total_budget_validates() -> None:
+    """The boundary is INCLUSIVE: exactly 8 000 000 characters across the
+    messages passes (13 at the 600 000 cap + one of 200 000), one
+    character more rejects."""
+
+    def _conversation(last_size: int) -> list[dict]:
+        return [{"who": "brain", "text": "a" * TEXT_CAP} for _ in range(13)] + [
+            {"who": "brain", "text": "b" * last_size}
+        ]
+
+    at = SavedChatCreate.model_validate({"messages": _conversation(200_000)})
+    assert sum(len(m.text) for m in at.messages) == TOTAL_BUDGET
+    with pytest.raises(ValidationError) as exc:
+        SavedChatCreate.model_validate({"messages": _conversation(200_001)})
+    assert "8,000,000-character budget" in exc.value.errors()[0]["msg"]
+
+
 # ---------------------------------------------------------------------------
 # JSON-shape regression: same KEYS as before, value bounds only
 # ---------------------------------------------------------------------------
@@ -747,7 +965,7 @@ def test_segments_absent_by_default_and_omitted_in_dump() -> None:

 def test_segments_thinking_entry_round_trips_and_caps() -> None:
    """The thinking entry: ``min_length=1`` (the zero-char run the
-    client never stores) + ``max_length=32_000`` (HistoryTurn's cap,
+    client never stores) + ``max_length=600_000`` (HistoryTurn's cap,
    phase 83) — at the cap validates, one past raises naming the
    offender; a missing ``text`` too."""
    at_cap = ChatMessage.model_validate(