docs(todo): log live testing findings L1-L7 (brain.reeseapps.com session)
Interactive browser test session (15 tests) against the deployed instance. Findings: L1 banner 'Try again' is plain text, L2 honesty gate eagerness + stochastic deflection compliance, L3 unstyled modal scrollbar, L5 recurring weak-hit 2nd source chip, L6 4000-char question clamp exceeds the embed input cap (500 surfaced as 'unreachable'), L7 save-as-doc drafts have no discard path. Each entry carries observed evidence, suggested fixes and acceptance.
This commit is contained in:
@@ -1 +1,209 @@
|
|||||||
# TODO
|
# TODO
|
||||||
|
|
||||||
|
## L1 — Chat error banner: "Try again" is plain text, not a button (2026-09-15, brain-of-reese interactive test)
|
||||||
|
|
||||||
|
`showErrorBanner()` (`frontend/assets/app.js`, `ERROR_HINT` ~line 362) renders
|
||||||
|
`${detail} ${ERROR_HINT}` as **plain text** into `#kb-banner-text`. The hint
|
||||||
|
begins with "Try again — if this persists, check the LLM is reachable.", so
|
||||||
|
"Try again" reads as a clickable action but is not — the only recovery is to
|
||||||
|
re-type the question. Observed on the "The chat model dropped the connection —
|
||||||
|
try again?" error after a mid-stream LLM drop (the per-answer Retry pill does
|
||||||
|
work; the banner does not).
|
||||||
|
|
||||||
|
Suggested fix: give the chat-view error banner a real Retry control that
|
||||||
|
re-asks the last question (the stale-chat banner already has the pattern:
|
||||||
|
`#stale-regenerate` → `retryLastTurn`; the phase-49 Retry pill asset exists).
|
||||||
|
Keep `#kb-banner` dual-use working for the RAG view, or split the two banners
|
||||||
|
if the RAG view's banner has different recovery semantics.
|
||||||
|
|
||||||
|
Acceptance: after a dropped/failed turn, the banner shows a visible Retry
|
||||||
|
button that re-runs the last question without re-typing; unit test for the
|
||||||
|
banner's button presence/handler in the frontend test suite.
|
||||||
|
|
||||||
|
## L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test)
|
||||||
|
|
||||||
|
Test: asked "What is the capital of Mongolia?" — known to the LLM, not in
|
||||||
|
the KB. Expected (README + locked HONESTY GATE): admit no notes, offer 2-3
|
||||||
|
alternative questions about things the brain DOES have notes on.
|
||||||
|
Observed: it answered "Ulaanbaatar" with a disclosure ("that info isn't in my
|
||||||
|
knowledge base… pulled that from general knowledge") and a generic
|
||||||
|
"happy to help" — and the turn carried **two unrelated source chips**
|
||||||
|
(`EverQuest_Quests_Connected/cabeast/Trooper_Nagraz.pl`,
|
||||||
|
`Trooper_Begzei.pl`) that read as citations under a non-grounded answer.
|
||||||
|
|
||||||
|
Three parts, one root-cause chain:
|
||||||
|
|
||||||
|
a) **Honesty gate too eager** (`app/api/chat.py` L272–277, A8 revised
|
||||||
|
2026-08-21): HIGH/grounded when `best_cosine >= threshold` OR
|
||||||
|
`fts_hits > 0`. A single weak FTS token hit (suspected: the token
|
||||||
|
"capital" inside a quest file) promotes a non-KB question into grounded
|
||||||
|
mode and injects two irrelevant top-docs into the HIGH prompt — which
|
||||||
|
then tempts the model into a parametric answer instead of deflection.
|
||||||
|
Consider: require FTS hits to be *meaningful* (e.g. rank threshold,
|
||||||
|
stopword/short-token exclusion, or cosine corroboration) before flipping
|
||||||
|
to HIGH; or run the LOW prompt when top-docs score below a usefulness bar.
|
||||||
|
|
||||||
|
b) **Model violates the locked prompt** (Rule 1 "Answer ONLY from the
|
||||||
|
provided document context", Rule 3 HONESTY GATE): with irrelevant docs
|
||||||
|
injected it answered from general knowledge. The disclosure is better UX
|
||||||
|
than silence, but the contract says no pretending to know + 2-3 concrete
|
||||||
|
alternative questions — decide whether to (i) tighten the prompt copy
|
||||||
|
("if the documents do not answer the question, treat it as LOW"),
|
||||||
|
(ii) amend the locked prompt via the plan to explicitly permit disclosed
|
||||||
|
general-knowledge answers, or (iii) treat the observed behavior as
|
||||||
|
acceptable and document it. Owner decision required — the prompt text
|
||||||
|
is locked verbatim (change through the plan, not here).
|
||||||
|
|
||||||
|
c) **Misleading chips on non-grounded answers** (`chat.py:252` —
|
||||||
|
`done.sources` = weak hits when deflected; by design, but visually a
|
||||||
|
citation). When the answer is disclosed general knowledge (or a deflected
|
||||||
|
turn), the chips should be visually de-emphasized / labeled (e.g. "nearby
|
||||||
|
docs I have, in case"), or omitted when the brain says it didn't use
|
||||||
|
them. At minimum: never render them as answer citations.
|
||||||
|
|
||||||
|
Follow-up (same day, chip one-tap re-run): the *identical* question
|
||||||
|
("What is the capital of Mongolia?"), same two weak-hit chips
|
||||||
|
(Trooper_Nagraz.pl + Trooper_Begzei.pl), fresh chat, no test tuning notes —
|
||||||
|
produced a **clean, textbook deflection**: no parametric answer, explicit
|
||||||
|
"I don't have any notes on…", three concrete alternative questions (all
|
||||||
|
accurate, incl. citing the owner's own tuning note as the deployments source
|
||||||
|
of truth). So with the same retrieved docs injected, HONESTY GATE compliance
|
||||||
|
is **stochastic** across runs (run 1: parametric answer; run 2: perfect
|
||||||
|
deflection). Implication for the fix direction: the deterministic lever is
|
||||||
|
the gate (don't inject irrelevant docs — part a), not prompt copy alone;
|
||||||
|
a small local model cannot be relied on to obey Rules 1/3 100% when handed
|
||||||
|
misleading context.
|
||||||
|
|
||||||
|
Plus a docs nit: the README still promises the exact deflection copy
|
||||||
|
*"I haven't done anything like that"* — the mandated deflection opening was
|
||||||
|
removed in the 2026-08-22 locked-prompt revision (`app/rag/prompts.py`
|
||||||
|
module docstring). Update the README's "If it doesn't have notes…" paragraph
|
||||||
|
to match current behavior.
|
||||||
|
|
||||||
|
Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports
|
||||||
|
results) produces no false citations, follows the HONESTY GATE (or the
|
||||||
|
amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a
|
||||||
|
weak single-token FTS hit.
|
||||||
|
|
||||||
|
## L3 — Document modal: native unstyled horizontal scrollbar in the code block (cosmetic) (2026-09-15, brain-of-reese interactive test)
|
||||||
|
|
||||||
|
In the document modal (the almost-fullscreen chip viewer), the raw-content
|
||||||
|
code block overflows horizontally and reveals the **browser-native, unstyled
|
||||||
|
scrollbar** (light-gray bar) — it clashes with the dark theme, and long
|
||||||
|
lines clip at the right edge (observed on a quest `.pl` file whose
|
||||||
|
`quest::say(...)` line exceeds the modal width). Style the scrollbar to
|
||||||
|
match the theme (`scrollbar-color` for Firefox, `::-webkit-scrollbar`
|
||||||
|
pseudos for Chromium), or consider wrapping long lines per content type.
|
||||||
|
|
||||||
|
Acceptance: the modal's code block scrolls horizontally with a themed
|
||||||
|
scrollbar; screenshot check in the document-viewer E2E story.
|
||||||
|
|
||||||
|
## L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)
|
||||||
|
|
||||||
|
Pattern across three live turns (owner-confirmed concern): the answer's
|
||||||
|
source chips always show **two** documents, and the second is frequently a
|
||||||
|
weak/adjacent hit the answer does not actually rely on — rendered with the
|
||||||
|
same visual weight as a real citation.
|
||||||
|
|
||||||
|
Observed cases:
|
||||||
|
|
||||||
|
1. "What is the capital of Mongolia?" (non-KB question) →
|
||||||
|
`EverQuest_Quests_Connected/cabeast/Trooper_Nagraz.pl` +
|
||||||
|
`Trooper_Begzei.pl` — both unrelated; the FTS token "capital" matched
|
||||||
|
the Iksar-Empire dialogue line in Nagraz (verified in the doc modal).
|
||||||
|
2. Phase-gate question (legit, answered from `brain-of-reese/.agents/validate.sh`) →
|
||||||
|
second chip `ServMon/README.md` — no mention of it in the answer.
|
||||||
|
3. Trooper_Nagraz question (legit, answered from `Trooper_Nagraz.pl`) →
|
||||||
|
second chip `Trooper_Byzin.pl` (a sibling NPC script) — not cited in the
|
||||||
|
answer.
|
||||||
|
4. Meta question about the conversation's own history ("What was the first
|
||||||
|
question I asked…", answered from chat history, no doc needed) → chips
|
||||||
|
`brain-of-reese/app/api/suggestions.py` +
|
||||||
|
`brain-of-reese/.agents/phases/complete/108_history_wire_check/00_phase.md`
|
||||||
|
— neither used.
|
||||||
|
|
||||||
|
Root cause chain: `top_n_docs = 2` (`app/config.py` L114) forces retrieval
|
||||||
|
to return two documents, `done.sources` carries both (`app/api/chat.py`
|
||||||
|
L252), and the UI chips every entry without distinguishing "the answer used
|
||||||
|
this" from "this also scored". Compounded by the L2 gate eagerness
|
||||||
|
(`fts_hits > 0 → HIGH`), which promotes weak token hits into the top-N.
|
||||||
|
|
||||||
|
Suggested direction (pick/combine, owner call):
|
||||||
|
|
||||||
|
* **Cite-gated chips** — the locked prompt Rule 1 already says "Cite which
|
||||||
|
document(s) you used, by path": parse the answer's cited paths and chip
|
||||||
|
only those (fall back to retrieval order when nothing is parseable);
|
||||||
|
drop or de-emphasize uncited retrievals.
|
||||||
|
* **Usefulness bar on the 2nd doc** — only include a document in
|
||||||
|
`done.sources` when its fused/cosine score clears a threshold (a single
|
||||||
|
weak FTS token hit should not earn a citation slot); `top_n_docs` stays
|
||||||
|
a ceiling, not a quota.
|
||||||
|
* **Visual split** — keep both, but render uncited/weak docs as a clearly
|
||||||
|
secondary "related docs" row, not citation chips.
|
||||||
|
|
||||||
|
Acceptance: for a single-document question, the turn shows one citation
|
||||||
|
chip; a unit test pins `done.sources` filtering (or the chip renderer's
|
||||||
|
cite-gate) on the four observed shapes; E2E asserts chip count for a
|
||||||
|
known single-source question.
|
||||||
|
|
||||||
|
## L6 — 4,000-char question clamp exceeds the embed model's input cap → misleading "couldn't reach the embedding model" error (2026-09-15, brain-of-reese interactive test)
|
||||||
|
|
||||||
|
Repro: type/paste a question to the UI maximum (the composer clamps at
|
||||||
|
4,000 chars — char counter shows "4000/4000 — character limit") and send.
|
||||||
|
Result, **100% reproducible**: the turn dies pre-token with the banner
|
||||||
|
"I couldn't reach the embedding model — please try again." + ERROR_HINT
|
||||||
|
"check the LLM is reachable" — both diagnoses are wrong.
|
||||||
|
|
||||||
|
Direct probe of the endpoint proves the real cause: a short question
|
||||||
|
embeds fine (HTTP 200), but the 4,000-char question (~903 tokens) gets
|
||||||
|
**HTTP 500** from aipi: `input (903 tokens) is too large to process.
|
||||||
|
increase the physical batch size (current batch size: 512)` (litellm).
|
||||||
|
So the maximum legal question length exceeds the embed model's maximum
|
||||||
|
legal input — and the chunker's own 1200-char cap (set to stay under the
|
||||||
|
~1024-token per-request cap) shows the question path never got the same
|
||||||
|
treatment.
|
||||||
|
|
||||||
|
Suggested fixes (pick/combine, owner call):
|
||||||
|
|
||||||
|
* **Truncate for embedding** — embed a bounded prefix of the question
|
||||||
|
(e.g. the same 1200-char budget as chunks) while the full question still
|
||||||
|
reaches the LLM prompt.
|
||||||
|
* **Map the 500 to a precise error** — detect the "too large" embed failure
|
||||||
|
and surface "question too long — trim it" (and fix the ERROR_HINT for
|
||||||
|
this case: reachability is fine).
|
||||||
|
* **Re-check the char clamp** — if the embed cap is ~512-1024 tokens and
|
||||||
|
worst-case tokenization is ~1.4 chars/token (chunker rationale), a clamp
|
||||||
|
well under 4,000 chars (or the truncation above) keeps every legal
|
||||||
|
question embeddable.
|
||||||
|
|
||||||
|
Acceptance: a 4,000-char question either succeeds (truncated embedding)
|
||||||
|
or fails with an accurate too-long error; unit test pins the error mapping;
|
||||||
|
the L1 "Try again" button fix should also apply to this banner.
|
||||||
|
|
||||||
|
## L7 — "Save as doc" has no Discard: orphan drafts are invisible and un-deletable (2026-09-15, brain-of-reese interactive test)
|
||||||
|
|
||||||
|
Clicking "Save as doc" on an answer POSTs `/api/doc-drafts` (201) and
|
||||||
|
navigates to `/doc-edit.html?draft=<token>`. The edit screen offers exactly
|
||||||
|
one action: **"Push to docs branch"**. There is no Discard/cancel control,
|
||||||
|
the drafts API has no DELETE route (`app/api/doc_drafts.py`: POST, GET,
|
||||||
|
PUT, POST /push only) and no TTL/pruning. A draft created by an accidental
|
||||||
|
click (or a tester) sits orphaned in the DB forever — invisible (no UI
|
||||||
|
lists drafts) and only consumable by actually pushing a doc to the repo.
|
||||||
|
|
||||||
|
Side observation (edge case, same test): the draft's default title is the
|
||||||
|
user record *immediately preceding* the saved answer. After a Retry
|
||||||
|
redo-in-place (L-see app.js `retryLastTurn`), the redone answer sits at the
|
||||||
|
end of the conversation, so its save-as-doc title came from an unrelated
|
||||||
|
trailing question (the doc was titled with a junk 4,000-char test question,
|
||||||
|
not the question the answer answered). Body is the full session
|
||||||
|
transcript (correct by design); title derivation is just the mismatch.
|
||||||
|
|
||||||
|
Suggested fix: add a Discard control to `/doc-edit.html` wired to a new
|
||||||
|
`DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's
|
||||||
|
credential), or a TTL/prune for stale drafts; consider deriving the default
|
||||||
|
title from the question the answer actually answered (its paired user
|
||||||
|
record) rather than the preceding record.
|
||||||
|
|
||||||
|
Acceptance: an orphaned draft can be discarded from the edit screen; the
|
||||||
|
draft row is gone afterward (API test); title of a save-as-doc after a
|
||||||
|
retry redo matches the redone answer's own question.
|
||||||
|
|||||||
Reference in New Issue
Block a user