From 2b75f3cc857b004f386456e07a83bf2cc1a02f4a Mon Sep 17 00:00:00 2001 From: ducoterra Date: Mon, 14 Sep 2026 21:31:08 -0400 Subject: [PATCH] docs(todo): log live testing findings L1-L7 (brain.reeseapps.com session) Interactive browser test session (15 tests) against the deployed instance. Findings: L1 banner 'Try again' is plain text, L2 honesty gate eagerness + stochastic deflection compliance, L3 unstyled modal scrollbar, L5 recurring weak-hit 2nd source chip, L6 4000-char question clamp exceeds the embed input cap (500 surfaced as 'unreachable'), L7 save-as-doc drafts have no discard path. Each entry carries observed evidence, suggested fixes and acceptance. --- TODO.md | 208 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/TODO.md b/TODO.md index 4640904..c1b0954 100644 --- a/TODO.md +++ b/TODO.md @@ -1 +1,209 @@ # TODO + +## L1 — Chat error banner: "Try again" is plain text, not a button (2026-09-15, brain-of-reese interactive test) + +`showErrorBanner()` (`frontend/assets/app.js`, `ERROR_HINT` ~line 362) renders +`${detail} ${ERROR_HINT}` as **plain text** into `#kb-banner-text`. The hint +begins with "Try again — if this persists, check the LLM is reachable.", so +"Try again" reads as a clickable action but is not — the only recovery is to +re-type the question. Observed on the "The chat model dropped the connection — +try again?" error after a mid-stream LLM drop (the per-answer Retry pill does +work; the banner does not). + +Suggested fix: give the chat-view error banner a real Retry control that +re-asks the last question (the stale-chat banner already has the pattern: +`#stale-regenerate` → `retryLastTurn`; the phase-49 Retry pill asset exists). +Keep `#kb-banner` dual-use working for the RAG view, or split the two banners +if the RAG view's banner has different recovery semantics. + +Acceptance: after a dropped/failed turn, the banner shows a visible Retry +button that re-runs the last question without re-typing; unit test for the +banner's button presence/handler in the frontend test suite. + +## L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test) + +Test: asked "What is the capital of Mongolia?" — known to the LLM, not in +the KB. Expected (README + locked HONESTY GATE): admit no notes, offer 2-3 +alternative questions about things the brain DOES have notes on. +Observed: it answered "Ulaanbaatar" with a disclosure ("that info isn't in my +knowledge base… pulled that from general knowledge") and a generic +"happy to help" — and the turn carried **two unrelated source chips** +(`EverQuest_Quests_Connected/cabeast/Trooper_Nagraz.pl`, +`Trooper_Begzei.pl`) that read as citations under a non-grounded answer. + +Three parts, one root-cause chain: + +a) **Honesty gate too eager** (`app/api/chat.py` L272–277, A8 revised + 2026-08-21): HIGH/grounded when `best_cosine >= threshold` OR + `fts_hits > 0`. A single weak FTS token hit (suspected: the token + "capital" inside a quest file) promotes a non-KB question into grounded + mode and injects two irrelevant top-docs into the HIGH prompt — which + then tempts the model into a parametric answer instead of deflection. + Consider: require FTS hits to be *meaningful* (e.g. rank threshold, + stopword/short-token exclusion, or cosine corroboration) before flipping + to HIGH; or run the LOW prompt when top-docs score below a usefulness bar. + +b) **Model violates the locked prompt** (Rule 1 "Answer ONLY from the + provided document context", Rule 3 HONESTY GATE): with irrelevant docs + injected it answered from general knowledge. The disclosure is better UX + than silence, but the contract says no pretending to know + 2-3 concrete + alternative questions — decide whether to (i) tighten the prompt copy + ("if the documents do not answer the question, treat it as LOW"), + (ii) amend the locked prompt via the plan to explicitly permit disclosed + general-knowledge answers, or (iii) treat the observed behavior as + acceptable and document it. Owner decision required — the prompt text + is locked verbatim (change through the plan, not here). + +c) **Misleading chips on non-grounded answers** (`chat.py:252` — + `done.sources` = weak hits when deflected; by design, but visually a + citation). When the answer is disclosed general knowledge (or a deflected + turn), the chips should be visually de-emphasized / labeled (e.g. "nearby + docs I have, in case"), or omitted when the brain says it didn't use + them. At minimum: never render them as answer citations. + +Follow-up (same day, chip one-tap re-run): the *identical* question +("What is the capital of Mongolia?"), same two weak-hit chips +(Trooper_Nagraz.pl + Trooper_Begzei.pl), fresh chat, no test tuning notes — +produced a **clean, textbook deflection**: no parametric answer, explicit +"I don't have any notes on…", three concrete alternative questions (all +accurate, incl. citing the owner's own tuning note as the deployments source +of truth). So with the same retrieved docs injected, HONESTY GATE compliance +is **stochastic** across runs (run 1: parametric answer; run 2: perfect +deflection). Implication for the fix direction: the deterministic lever is +the gate (don't inject irrelevant docs — part a), not prompt copy alone; +a small local model cannot be relied on to obey Rules 1/3 100% when handed +misleading context. + +Plus a docs nit: the README still promises the exact deflection copy +*"I haven't done anything like that"* — the mandated deflection opening was +removed in the 2026-08-22 locked-prompt revision (`app/rag/prompts.py` +module docstring). Update the README's "If it doesn't have notes…" paragraph +to match current behavior. + +Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports +results) produces no false citations, follows the HONESTY GATE (or the +amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a +weak single-token FTS hit. + +## L3 — Document modal: native unstyled horizontal scrollbar in the code block (cosmetic) (2026-09-15, brain-of-reese interactive test) + +In the document modal (the almost-fullscreen chip viewer), the raw-content +code block overflows horizontally and reveals the **browser-native, unstyled +scrollbar** (light-gray bar) — it clashes with the dark theme, and long +lines clip at the right edge (observed on a quest `.pl` file whose +`quest::say(...)` line exceeds the modal width). Style the scrollbar to +match the theme (`scrollbar-color` for Firefox, `::-webkit-scrollbar` +pseudos for Chromium), or consider wrapping long lines per content type. + +Acceptance: the modal's code block scrolls horizontally with a themed +scrollbar; screenshot check in the document-viewer E2E story. + +## L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test) + +Pattern across three live turns (owner-confirmed concern): the answer's +source chips always show **two** documents, and the second is frequently a +weak/adjacent hit the answer does not actually rely on — rendered with the +same visual weight as a real citation. + +Observed cases: + +1. "What is the capital of Mongolia?" (non-KB question) → + `EverQuest_Quests_Connected/cabeast/Trooper_Nagraz.pl` + + `Trooper_Begzei.pl` — both unrelated; the FTS token "capital" matched + the Iksar-Empire dialogue line in Nagraz (verified in the doc modal). +2. Phase-gate question (legit, answered from `brain-of-reese/.agents/validate.sh`) → + second chip `ServMon/README.md` — no mention of it in the answer. +3. Trooper_Nagraz question (legit, answered from `Trooper_Nagraz.pl`) → + second chip `Trooper_Byzin.pl` (a sibling NPC script) — not cited in the + answer. +4. Meta question about the conversation's own history ("What was the first + question I asked…", answered from chat history, no doc needed) → chips + `brain-of-reese/app/api/suggestions.py` + + `brain-of-reese/.agents/phases/complete/108_history_wire_check/00_phase.md` + — neither used. + +Root cause chain: `top_n_docs = 2` (`app/config.py` L114) forces retrieval +to return two documents, `done.sources` carries both (`app/api/chat.py` +L252), and the UI chips every entry without distinguishing "the answer used +this" from "this also scored". Compounded by the L2 gate eagerness +(`fts_hits > 0 → HIGH`), which promotes weak token hits into the top-N. + +Suggested direction (pick/combine, owner call): + +* **Cite-gated chips** — the locked prompt Rule 1 already says "Cite which + document(s) you used, by path": parse the answer's cited paths and chip + only those (fall back to retrieval order when nothing is parseable); + drop or de-emphasize uncited retrievals. +* **Usefulness bar on the 2nd doc** — only include a document in + `done.sources` when its fused/cosine score clears a threshold (a single + weak FTS token hit should not earn a citation slot); `top_n_docs` stays + a ceiling, not a quota. +* **Visual split** — keep both, but render uncited/weak docs as a clearly + secondary "related docs" row, not citation chips. + +Acceptance: for a single-document question, the turn shows one citation +chip; a unit test pins `done.sources` filtering (or the chip renderer's +cite-gate) on the four observed shapes; E2E asserts chip count for a +known single-source question. + +## L6 — 4,000-char question clamp exceeds the embed model's input cap → misleading "couldn't reach the embedding model" error (2026-09-15, brain-of-reese interactive test) + +Repro: type/paste a question to the UI maximum (the composer clamps at +4,000 chars — char counter shows "4000/4000 — character limit") and send. +Result, **100% reproducible**: the turn dies pre-token with the banner +"I couldn't reach the embedding model — please try again." + ERROR_HINT +"check the LLM is reachable" — both diagnoses are wrong. + +Direct probe of the endpoint proves the real cause: a short question +embeds fine (HTTP 200), but the 4,000-char question (~903 tokens) gets +**HTTP 500** from aipi: `input (903 tokens) is too large to process. +increase the physical batch size (current batch size: 512)` (litellm). +So the maximum legal question length exceeds the embed model's maximum +legal input — and the chunker's own 1200-char cap (set to stay under the +~1024-token per-request cap) shows the question path never got the same +treatment. + +Suggested fixes (pick/combine, owner call): + +* **Truncate for embedding** — embed a bounded prefix of the question + (e.g. the same 1200-char budget as chunks) while the full question still + reaches the LLM prompt. +* **Map the 500 to a precise error** — detect the "too large" embed failure + and surface "question too long — trim it" (and fix the ERROR_HINT for + this case: reachability is fine). +* **Re-check the char clamp** — if the embed cap is ~512-1024 tokens and + worst-case tokenization is ~1.4 chars/token (chunker rationale), a clamp + well under 4,000 chars (or the truncation above) keeps every legal + question embeddable. + +Acceptance: a 4,000-char question either succeeds (truncated embedding) +or fails with an accurate too-long error; unit test pins the error mapping; +the L1 "Try again" button fix should also apply to this banner. + +## L7 — "Save as doc" has no Discard: orphan drafts are invisible and un-deletable (2026-09-15, brain-of-reese interactive test) + +Clicking "Save as doc" on an answer POSTs `/api/doc-drafts` (201) and +navigates to `/doc-edit.html?draft=`. The edit screen offers exactly +one action: **"Push to docs branch"**. There is no Discard/cancel control, +the drafts API has no DELETE route (`app/api/doc_drafts.py`: POST, GET, +PUT, POST /push only) and no TTL/pruning. A draft created by an accidental +click (or a tester) sits orphaned in the DB forever — invisible (no UI +lists drafts) and only consumable by actually pushing a doc to the repo. + +Side observation (edge case, same test): the draft's default title is the +user record *immediately preceding* the saved answer. After a Retry +redo-in-place (L-see app.js `retryLastTurn`), the redone answer sits at the +end of the conversation, so its save-as-doc title came from an unrelated +trailing question (the doc was titled with a junk 4,000-char test question, +not the question the answer answered). Body is the full session +transcript (correct by design); title derivation is just the mismatch. + +Suggested fix: add a Discard control to `/doc-edit.html` wired to a new +`DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's +credential), or a TTL/prune for stale drafts; consider deriving the default +title from the question the answer actually answered (its paired user +record) rather than the preceding record. + +Acceptance: an orphaned draft can be discarded from the edit screen; the +draft row is gone afterward (API test); title of a save-as-doc after a +retry redo matches the redone answer's own question.