brain-of-reese/TODO.md

TODO

L1 — Chat error banner: "Try again" is plain text, not a button (2026-09-15, brain-of-reese interactive test)

showErrorBanner() (frontend/assets/app.js, ERROR_HINT ~line 362) renders ${detail} ${ERROR_HINT} as plain text into #kb-banner-text. The hint begins with "Try again — if this persists, check the LLM is reachable.", so "Try again" reads as a clickable action but is not — the only recovery is to re-type the question. Observed on the "The chat model dropped the connection — try again?" error after a mid-stream LLM drop (the per-answer Retry pill does work; the banner does not).

Suggested fix: give the chat-view error banner a real Retry control that re-asks the last question (the stale-chat banner already has the pattern: #stale-regenerate → retryLastTurn; the phase-49 Retry pill asset exists). Keep #kb-banner dual-use working for the RAG view, or split the two banners if the RAG view's banner has different recovery semantics.

Acceptance: after a dropped/failed turn, the banner shows a visible Retry button that re-runs the last question without re-typing; unit test for the banner's button presence/handler in the frontend test suite.

L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test)

Test: asked "What is the capital of Mongolia?" — known to the LLM, not in the KB. Expected (README + locked HONESTY GATE): admit no notes, offer 2-3 alternative questions about things the brain DOES have notes on. Observed: it answered "Ulaanbaatar" with a disclosure ("that info isn't in my knowledge base… pulled that from general knowledge") and a generic "happy to help" — and the turn carried two unrelated source chips (EverQuest_Quests_Connected/cabeast/Trooper_Nagraz.pl, Trooper_Begzei.pl) that read as citations under a non-grounded answer.

Three parts, one root-cause chain:

a) Honesty gate too eager (app/api/chat.py L272–277, A8 revised 2026-08-21): HIGH/grounded when best_cosine >= threshold OR fts_hits > 0. A single weak FTS token hit (suspected: the token "capital" inside a quest file) promotes a non-KB question into grounded mode and injects two irrelevant top-docs into the HIGH prompt — which then tempts the model into a parametric answer instead of deflection. Consider: require FTS hits to be meaningful (e.g. rank threshold, stopword/short-token exclusion, or cosine corroboration) before flipping to HIGH; or run the LOW prompt when top-docs score below a usefulness bar.

b) Model violates the locked prompt (Rule 1 "Answer ONLY from the provided document context", Rule 3 HONESTY GATE): with irrelevant docs injected it answered from general knowledge. The disclosure is better UX than silence, but the contract says no pretending to know + 2-3 concrete alternative questions — decide whether to (i) tighten the prompt copy ("if the documents do not answer the question, treat it as LOW"), (ii) amend the locked prompt via the plan to explicitly permit disclosed general-knowledge answers, or (iii) treat the observed behavior as acceptable and document it. Owner decision required — the prompt text is locked verbatim (change through the plan, not here).

c) Misleading chips on non-grounded answers (chat.py:252 — done.sources = weak hits when deflected; by design, but visually a citation). When the answer is disclosed general knowledge (or a deflected turn), the chips should be visually de-emphasized / labeled (e.g. "nearby docs I have, in case"), or omitted when the brain says it didn't use them. At minimum: never render them as answer citations.

Follow-up (same day, chip one-tap re-run): the identical question ("What is the capital of Mongolia?"), same two weak-hit chips (Trooper_Nagraz.pl + Trooper_Begzei.pl), fresh chat, no test tuning notes — produced a clean, textbook deflection: no parametric answer, explicit "I don't have any notes on…", three concrete alternative questions (all accurate, incl. citing the owner's own tuning note as the deployments source of truth). So with the same retrieved docs injected, HONESTY GATE compliance is stochastic across runs (run 1: parametric answer; run 2: perfect deflection). Implication for the fix direction: the deterministic lever is the gate (don't inject irrelevant docs — part a), not prompt copy alone; a small local model cannot be relied on to obey Rules 1/3 100% when handed misleading context.

Plus a docs nit: the README still promises the exact deflection copy "I haven't done anything like that" — the mandated deflection opening was removed in the 2026-08-22 locked-prompt revision (app/rag/prompts.py module docstring). Update the README's "If it doesn't have notes…" paragraph to match current behavior.

Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports results) produces no false citations, follows the HONESTY GATE (or the amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a weak single-token FTS hit.

L3 — Document modal: native unstyled horizontal scrollbar in the code block (cosmetic) (2026-09-15, brain-of-reese interactive test)

In the document modal (the almost-fullscreen chip viewer), the raw-content code block overflows horizontally and reveals the browser-native, unstyled scrollbar (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge (observed on a quest .pl file whose quest::say(...) line exceeds the modal width). Style the scrollbar to match the theme (scrollbar-color for Firefox, ::-webkit-scrollbar pseudos for Chromium), or consider wrapping long lines per content type.

Acceptance: the modal's code block scrolls horizontally with a themed scrollbar; screenshot check in the document-viewer E2E story.

L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)

Pattern across three live turns (owner-confirmed concern): the answer's source chips always show two documents, and the second is frequently a weak/adjacent hit the answer does not actually rely on — rendered with the same visual weight as a real citation.

Observed cases:

1. "What is the capital of Mongolia?" (non-KB question) → EverQuest_Quests_Connected/cabeast/Trooper_Nagraz.pl + Trooper_Begzei.pl — both unrelated; the FTS token "capital" matched the Iksar-Empire dialogue line in Nagraz (verified in the doc modal).
2. Phase-gate question (legit, answered from brain-of-reese/.agents/validate.sh) → second chip ServMon/README.md — no mention of it in the answer.
3. Trooper_Nagraz question (legit, answered from Trooper_Nagraz.pl) → second chip Trooper_Byzin.pl (a sibling NPC script) — not cited in the answer.
4. Meta question about the conversation's own history ("What was the first question I asked…", answered from chat history, no doc needed) → chips brain-of-reese/app/api/suggestions.py + brain-of-reese/.agents/phases/complete/108_history_wire_check/00_phase.md — neither used.

Root cause chain: top_n_docs = 2 (app/config.py L114) forces retrieval to return two documents, done.sources carries both (app/api/chat.py L252), and the UI chips every entry without distinguishing "the answer used this" from "this also scored". Compounded by the L2 gate eagerness (fts_hits > 0 → HIGH), which promotes weak token hits into the top-N.

Suggested direction (pick/combine, owner call):

• Cite-gated chips — the locked prompt Rule 1 already says "Cite which document(s) you used, by path": parse the answer's cited paths and chip only those (fall back to retrieval order when nothing is parseable); drop or de-emphasize uncited retrievals.
• Usefulness bar on the 2nd doc — only include a document in done.sources when its fused/cosine score clears a threshold (a single weak FTS token hit should not earn a citation slot); top_n_docs stays a ceiling, not a quota.
• Visual split — keep both, but render uncited/weak docs as a clearly secondary "related docs" row, not citation chips.

Acceptance: for a single-document question, the turn shows one citation chip; a unit test pins done.sources filtering (or the chip renderer's cite-gate) on the four observed shapes; E2E asserts chip count for a known single-source question.

L6 — 4,000-char question clamp exceeds the embed model's input cap → misleading "couldn't reach the embedding model" error (2026-09-15, brain-of-reese interactive test)

Repro: type/paste a question to the UI maximum (the composer clamps at 4,000 chars — char counter shows "4000/4000 — character limit") and send. Result, 100% reproducible: the turn dies pre-token with the banner "I couldn't reach the embedding model — please try again." + ERROR_HINT "check the LLM is reachable" — both diagnoses are wrong.

Direct probe of the endpoint proves the real cause: a short question embeds fine (HTTP 200), but the 4,000-char question (~903 tokens) gets HTTP 500 from aipi: input (903 tokens) is too large to process. increase the physical batch size (current batch size: 512) (litellm). So the maximum legal question length exceeds the embed model's maximum legal input — and the chunker's own 1200-char cap (set to stay under the ~1024-token per-request cap) shows the question path never got the same treatment.

Suggested fixes (pick/combine, owner call):

• Truncate for embedding — embed a bounded prefix of the question (e.g. the same 1200-char budget as chunks) while the full question still reaches the LLM prompt.
• Map the 500 to a precise error — detect the "too large" embed failure and surface "question too long — trim it" (and fix the ERROR_HINT for this case: reachability is fine).
• Re-check the char clamp — if the embed cap is ~512-1024 tokens and worst-case tokenization is ~1.4 chars/token (chunker rationale), a clamp well under 4,000 chars (or the truncation above) keeps every legal question embeddable.

Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 "Try again" button fix should also apply to this banner.

L7 — "Save as doc" has no Discard: orphan drafts are invisible and un-deletable (2026-09-15, brain-of-reese interactive test)

Clicking "Save as doc" on an answer POSTs /api/doc-drafts (201) and navigates to /doc-edit.html?draft=<token>. The edit screen offers exactly one action: "Push to docs branch". There is no Discard/cancel control, the drafts API has no DELETE route (app/api/doc_drafts.py: POST, GET, PUT, POST /push only) and no TTL/pruning. A draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo.

Side observation (edge case, same test): the draft's default title is the user record immediately preceding the saved answer. After a Retry redo-in-place (L-see app.js retryLastTurn), the redone answer sits at the end of the conversation, so its save-as-doc title came from an unrelated trailing question (the doc was titled with a junk 4,000-char test question, not the question the answer answered). Body is the full session transcript (correct by design); title derivation is just the mismatch.

Suggested fix: add a Discard control to /doc-edit.html wired to a new DELETE /api/doc-drafts/{token} (admin-gated, token = the screen's credential), or a TTL/prune for stale drafts; consider deriving the default title from the question the answer actually answered (its paired user record) rather than the preceding record.

Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test); title of a save-as-doc after a retry redo matches the redone answer's own question.