Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
ducoterra
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/unit/test_source_chip_quality.py
+121 -94
View File
@@ -1,13 +1,15 @@
"""Unit: the phase-113 source-chip-quality contract (TODO L5 + L2c —
"the 2nd chip is often noise the answer never used").
Phase 113 demotes sub-floor hits out of the citation surface: the
done frame carries the cited tier in ``sources`` (rendered by
``appendSources`` as ``.source-chip`` pills, UNCHANGED) and the
related tier in ``related`` (rendered by the NEW ``appendRelated`` as
the de-emphasized labeled row — ``.related-doc`` links, never
``.source-chip``). A deflected turn carries ``sources: []`` → zero
chips; its weak hits live in the related row only.
Phase 118 re-tiers the same frame (LOCKED A3/A4): ``sources`` carries
the suggested tier (top-5, NO floor) + the agent-read docs (deduped)
— rendered by ``appendSources`` as ``.source-chip`` pills, UNCHANGED —
and ``related`` carries rank 6+ after the suggested set (rendered by
``appendRelated`` as the de-emphasized labeled row — ``.related-doc``
links, never ``.source-chip``). A deflected turn carries
``sources: []`` → zero chips; its weak hits are the suggested tier
(the durable record), and with ≤5 retrieved docs the related row is
empty.
This module pins the STATIC SOURCES the UI contract stands on, in the
house source-pin pattern (the test_chip_sizing_question_cap.py
@@ -26,24 +28,23 @@ house source-pin pattern (the test_chip_sizing_question_cap.py
1. **both docs weak** ("What is the capital of Mongolia?" →
``Trooper_Nagraz.pl`` + ``Trooper_Begzei.pl``, both unrelated) —
cited tier empty, the weak hits ride the related tier, capped at
``related_max_docs``; the FTS hit without vector corroboration
stays LOW (the A8-revised "Mongolia" case);
the weak hits are SUGGESTED (no floor, A3; the durable record),
nothing left for the related tier; the FTS hit without vector
corroboration stays LOW (the A8-revised "Mongolia" case);
2. **one strong + one weak** (the phase-gate question answered from
``brain-of-reese/.agents/validate.sh``; the 2nd chip
``ServMon/README.md`` unused) — exactly ONE cited ref, the weak
doc in ``related``;
``brain-of-reese/.agents/validate.sh``; ``ServMon/README.md``
alongside) — BOTH suggested (no floor) ⇒ both cited refs (A4),
no related tier;
3. **the Nagraz case** (``Trooper_Nagraz.pl`` strong,
``Trooper_Byzin.pl`` weak — same shape, different fixtures);
4. **the meta/history question** (no doc clears the bar, the agent
reads nothing — chips ``app/api/suggestions.py`` +
``108_history_wire_check/00_phase.md``, neither used) — pinned on
the DONE FRAME (endpoint-level, fake retriever/LLM/session): the
frame is row-only — ``sources: []`` (the UI's chip list — zero
chips) + the weak hits in ``related``;
5. **the agent-read exemption** (LOCKED A2): a below-floor doc the
agent ``read`` via the tool joins ``sources`` (cited, last) and is
excluded from ``related``.
4. **the meta/history question** (the agent reads nothing —
``app/api/suggestions.py`` + ``108_history_wire_check/00_phase.md``
alongside) — pinned on the DONE FRAME (endpoint-level, fake
retriever/LLM/session): ``sources: []`` (zero chips) and an EMPTY
related row (both weak docs are suggested, ≤5 docs retrieved);
5. **the agent-read exemption** (LOCKED A4): a rank-6+ (related-
tier) doc the agent ``read`` via the tool joins ``sources``
(cited, last) and is excluded from ``related``.
The browser behavior (chip counts on a single-source question, zero
chips on a deflected turn) is E2E-gated by
@@ -411,14 +412,18 @@ def _shape_settings() -> Settings:
"""The PRODUCTION calibration (the code defaults, explicit) — the
four shapes were observed LIVE under this threshold/floor pair.
``_env_file=None`` keeps the mock-calibrated values from
``tests/conftest.py`` (and any local ``.env``) out of the pin."""
``tests/conftest.py`` (and any local ``.env``) out of the pin.
``source_usefulness_floor`` / ``top_n_docs`` are legacy phase-113
settings — phase 118 retired their seeding role (A6; ``plan_turn``
never consults them), they are carried here for completeness."""
return Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=0.62,
lexical_support_floor=0.35,
source_usefulness_floor=0.35, # LOCKED A2 default
related_max_docs=2, # LOCKED A4 default
top_n_docs=2, # the ceiling — never a quota (LOCKED A2)
source_usefulness_floor=0.35, # retired by phase 118 (A6) — not consulted
related_max_docs=2, # the rank-6+ row cap (LOCKED A4)
top_n_docs=2, # retired by phase 118 (A6) — not consulted
suggested_docs=5, # the "start here" cap (LOCKED A3)
)
@@ -461,12 +466,12 @@ def _chunk(
def test_shape_1_mongolia_both_docs_weak_cite_nothing() -> None:
"""Observed shape 1 (TODO L110–113): "What is the capital of
Mongolia?" → chips ``Trooper_Nagraz.pl`` + ``Trooper_Begzei.pl``,
BOTH unrelated. Both below the bar: the cited tier is EMPTY (zero
citation chips) and the weak hits ride the related tier — in rank
order, capped at ``related_max_docs`` (the 3rd weak doc drops out).
The FTS hit without vector corroboration (0.20 < the 0.35 lexical
floor) stays LOW — the A8-revised "Mongolia" case; the weak content
never reaches the LLM."""
BOTH unrelated. Phase 118 (A3): the floor never filters — the weak
hits are the SUGGESTED tier (the durable record's input), and with
three retrieved docs nothing is left for the related tier (rank
6+). The FTS hit without vector corroboration (0.20 < the 0.35
lexical floor) stays LOW — the A8-revised "Mongolia" case; the weak
content never reaches the LLM (titles only)."""
nagraz = _doc("scripts", "scripts/Trooper_Nagraz.pl", "Trooper_Nagraz.pl",
"NAGRAZ_PL_CONTENT")
begzei = _doc("scripts", "scripts/Trooper_Begzei.pl", "Trooper_Begzei.pl",
@@ -476,64 +481,76 @@ def test_shape_1_mongolia_both_docs_weak_cite_nothing() -> None:
chunks = [
_chunk(nagraz, 0.033, cosine=0.20, fts_hit=True), # rank 1, lexical hit
_chunk(begzei, 0.031, cosine=0.12),
_chunk(third, 0.030, cosine=0.10), # below the cap — related drops it
_chunk(third, 0.030, cosine=0.10),
]
plan = chat_api.plan_turn(chunks, _shape_settings())
assert plan.deflected is True # 0.20 < 0.62 AND 0.20 < the 0.35 lex floor
assert plan.docs == [] # NO citation slot below the bar
assert [d.title for d in plan.related_docs] == [
# No floor (A3): the weak hits are suggested, in rank order (≤5).
assert [d.title for d in plan.suggested_docs] == [
"Trooper_Nagraz.pl",
"Trooper_Begzei.pl",
] # rank order, capped at related_max_docs (2)
assert len(plan.related_docs) <= 2
"Trooper_Third.pl",
]
assert plan.related_docs == [] # no rank-6+ doc among 3 retrieved
# The LOW prompt is titles only — none of the weak content is sent.
assert "NAGRAZ_PL_CONTENT" not in plan.system_prompt
assert "Trooper_Nagraz.pl" in plan.system_prompt # weak-hit titles carried
assert plan.suggestions # the "Maybe try" chips are unchanged
def test_shape_2_validate_sh_strong_plus_unused_second_chip() -> None:
def test_shape_2_validate_sh_strong_plus_weak_second_suggested() -> None:
"""Observed shape 2 (TODO L114–116): the phase-gate question is
answered from ``brain-of-reese/.agents/validate.sh`` — the 2nd chip
``ServMon/README.md`` was NEVER used. The strong doc clears the bar
and takes the only cited slot (top_n_docs is a ceiling, not a
quota); the weak 2nd doc demotes to related — never a citation.
The HIGH prompt carries the cited content only."""
answered from ``brain-of-reese/.agents/validate.sh`` with
``ServMon/README.md`` retrieved alongside (weak cosine). Phase 118
(A3): the floor never filters — the weak 2nd doc is SUGGESTED too
(both docs seed the HIGH prompt as summaries; the A5 fallback
carries the short fixture content whole), and A4 makes both
citation refs on the done frame — the "unused 2nd chip" is the
phase-113 shape, retired by the owner directive."""
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
"VALIDATE_SH_CONTENT")
servmon = _doc("ServMon", "README.md", "ServMon README",
"SERVMON_README_CONTENT")
chunks = [
_chunk(validate, 0.90, cosine=0.70), # clears threshold AND bar
_chunk(validate, 0.90, cosine=0.70), # clears the threshold
_chunk(servmon, 0.80, cosine=0.20), # high fused rank, weak cosine
]
plan = chat_api.plan_turn(chunks, _shape_settings())
assert plan.deflected is False # 0.70 >= 0.62
assert [d.title for d in plan.docs] == ["validate.sh"] # exactly ONE cited
assert [d.title for d in plan.related_docs] == ["ServMon README"]
assert "VALIDATE_SH_CONTENT" in plan.system_prompt
assert "SERVMON_README_CONTENT" not in plan.system_prompt
assert [d.title for d in plan.suggested_docs] == [
"validate.sh",
"ServMon README",
] # both suggested (no floor), rank order
assert plan.related_docs == [] # nothing beyond rank 2 for 2 docs
assert "VALIDATE_SH_CONTENT" in plan.system_prompt # A5 preview fallback
assert "SERVMON_README_CONTENT" in plan.system_prompt # ditto
def test_shape_3_nagraz_answered_by_own_doc_byzin_uncited() -> None:
def test_shape_3_nagraz_answered_by_own_doc_byzin_suggested() -> None:
"""Observed shape 3 (TODO L117–119): the Trooper_Nagraz question is
answered from ``Trooper_Nagraz.pl`` — the 2nd chip
``Trooper_Byzin.pl`` uncited. The SAME shape as shape 2 with
different fixtures — the bar filters the 2nd chip; it is not a
coincidence of the validate.sh pair."""
answered from ``Trooper_Nagraz.pl`` with ``Trooper_Byzin.pl``
retrieved alongside (weak cosine). The SAME shape as shape 2 with
different fixtures — phase 118's no-floor tiering suggests BOTH
(the phase-113 "bar filters the 2nd chip" story is retired); the
HIGH prompt seeds both summaries (A5 fallback for the short
fixture content)."""
nagraz = _doc("scripts", "scripts/Trooper_Nagraz.pl", "Trooper_Nagraz.pl",
"NAGRAZ_PL_CONTENT")
byzin = _doc("scripts", "scripts/Trooper_Byzin.pl", "Trooper_Byzin.pl",
"BYZIN_PL_CONTENT")
chunks = [
_chunk(nagraz, 0.85, cosine=0.70),
_chunk(byzin, 0.75, cosine=0.15), # below the bar
_chunk(byzin, 0.75, cosine=0.15), # weak cosine — still suggested (A3)
]
plan = chat_api.plan_turn(chunks, _shape_settings())
assert plan.deflected is False
assert [d.title for d in plan.docs] == ["Trooper_Nagraz.pl"] # 1 cited
assert [d.title for d in plan.related_docs] == ["Trooper_Byzin.pl"] # 1 related
assert "BYZIN_PL_CONTENT" not in plan.system_prompt
assert [d.title for d in plan.suggested_docs] == [
"Trooper_Nagraz.pl",
"Trooper_Byzin.pl",
] # both suggested (no floor), rank order
assert plan.related_docs == []
assert "NAGRAZ_PL_CONTENT" in plan.system_prompt # A5 preview fallback
assert "BYZIN_PL_CONTENT" in plan.system_prompt # ditto
# ---------- task 03: done-frame wire (endpoint-level fakes, no stack) ----------
@@ -664,21 +681,22 @@ def _fake_retriever(chunks: list[RetrievedChunk]) -> Any:
return retrieve
def test_shape_4_meta_question_deflected_frame_is_row_only(
def test_shape_4_meta_question_deflected_frame_has_no_chips_or_row(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Observed shape 4 (TODO L120–123), pinned on the DONE FRAME: a
meta question about the conversation's own history → chips
meta question about the conversation's own history → weak hits
``app/api/suggestions.py`` + ``108_history_wire_check/00_phase.md``,
neither used. No doc clears the bar and the agent reads nothing —
the frame is ROW-ONLY: ``sources: []`` (the UI chips every source
entry — zero chips) with the weak hits in ``related`` (rank order,
≤ ``related_max_docs``) — the de-emphasized row's links (the row's
rendering itself is pinned by task 02's source tests + the E2E).
The weak retrieval stays durably recorded (LOCKED A3); the weak
content never reaches the LLM (LOW prompt, titles only)."""
neither used. Phase 118: the agent reads nothing, the weak hits
are the SUGGESTED tier (no floor, A3 — the durable record's input)
and, with only two retrieved docs, nothing reaches rank 6+ — the
frame carries ``sources: []`` (zero chips — a deflected answer
cites nothing) AND an empty ``related`` row (the row's rendering
itself is pinned by task 02's source tests + the E2E). The weak
retrieval stays durably recorded (LOCKED A3); the weak content
never reaches the LLM (LOW prompt, titles only)."""
session, llm = chip_env
suggestions = _doc("brain-of-reese", "app/api/suggestions.py",
"suggestions.py", "SUGGESTIONS_PY_CONTENT")
@@ -701,13 +719,9 @@ def test_shape_4_meta_question_deflected_frame_is_row_only(
assert done["type"] == "done"
assert done["deflected"] is True
assert done["sources"] == [] # zero citation chips on the wire
related = done["related"]
assert [(s["source"], s["path"]) for s in related] == [
("brain-of-reese", "app/api/suggestions.py"),
("brain-of-reese", ".agents/108_history_wire_check/00_phase.md"),
] # rank order
assert len(related) <= 2 # related_max_docs
assert all(s["title"] for s in related) # the row's links carry the identity
# Phase 118 (A3): both weak docs are suggested (≤5, no floor) —
# nothing reaches rank 6+, so the related row is empty.
assert done["related"] == []
assert done["suggestions"] # the "Maybe try" chips are unchanged
(system, _user) = llm.seen[0][0], llm.seen[0][1]
@@ -722,16 +736,17 @@ def test_shape_4_meta_question_deflected_frame_is_row_only(
assert "108_history_wire_check/00_phase.md" in row.sources
def test_done_frame_single_cited_ref_strong_plus_weak(
def test_done_frame_carries_suggested_refs_strong_plus_weak(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Shape 2 on the wire — the single-document question's input to
"exactly one citation chip" (the E2E asserts the rendered chip):
the bar-clearing doc is the ONLY ``sources`` ref; the weak 2nd doc
rides ``related``; the tiers are disjoint (the done frame's dedupe).
The durable record keeps the FULL retrieval (LOCKED A3)."""
"""Shape 2 on the wire under phase 118 (LOCKED A4): the citation
surface is the suggested tier + the agent's reads (deduped) —
with two retrieved docs and no read, BOTH docs are ``sources``
refs (no floor — A3); nothing reaches rank 6+, so ``related`` is
empty; the tiers stay disjoint (the done frame's dedupe). The
durable record keeps the FULL retrieval (LOCKED A3)."""
session, _llm = chip_env
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
"VALIDATE_SH_CONTENT")
@@ -753,10 +768,9 @@ def test_done_frame_single_cited_ref_strong_plus_weak(
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("brain-of-reese", ".agents/validate.sh"),
] # EXACTLY one citation chip on the wire
assert [(s["source"], s["path"]) for s in done["related"]] == [
("ServMon", "README.md"),
("ServMon", "README.md"), # A4: suggested + read — both suggested (A3)
]
assert done["related"] == [] # nothing reaches rank 6+ for 2 docs
cited = {(s["source"], s["path"]) for s in done["sources"]}
related = {(s["source"], s["path"]) for s in done["related"]}
assert cited.isdisjoint(related)
@@ -769,20 +783,25 @@ def test_done_frame_single_cited_ref_strong_plus_weak(
assert "ServMon/README.md" in row.sources
def test_agent_read_below_floor_doc_joins_sources(
def test_agent_read_related_doc_is_cited_not_related(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The agent-read exemption (LOCKED A2): a doc UNDER the bar that
the agent ``read`` via the tool is cited by definition — the model
read it, so it was used. It joins ``sources`` (after the retrieved
cited docs, deduped) and is EXCLUDED from ``related`` (a used doc
must never read as "nearby"); the other below-floor doc stays in
the tier. The read content reached the model (the tool result in
the follow-up request)."""
"""The agent-read exemption (LOCKED A4, phase-118 tiering): a
rank-6+ doc — the related tier ("nearby docs") — that the agent
``read`` via the tool is cited by definition: the model read it, so
it was used. It joins ``sources`` (after the suggested docs — it
was not suggested, so the read appends it last, deduped) and is
EXCLUDED from ``related`` (a used doc must never read as "nearby");
the other rank-6+ doc stays in the tier. The read content reached
the model (the tool result in the follow-up request)."""
session, _default_llm = chip_env
strong = _doc("docs", "strong.md", "Strong", "STRONG_DOC_CONTENT")
fillers = [
_doc("docs", f"filler{i}.md", f"Filler {i}", f"FILLER_{i}_CONTENT")
for i in range(1, 5) # ranks 2–5 — fill the suggested tier
]
weak_b = _doc("docs", "weak-b.md", "Weak B", "WEAK_B_READ_BY_AGENT")
weak_c = _doc("docs", "weak-c.md", "Weak C", "WEAK_C_CONTENT")
monkeypatch.setattr(
@@ -790,9 +809,13 @@ def test_agent_read_below_floor_doc_joins_sources(
"retrieve",
_fake_retriever(
[
_chunk(strong, 0.90, cosine=0.70), # clears the bar
_chunk(weak_b, 0.80, cosine=0.20), # below the bar — read by the agent
_chunk(weak_c, 0.70, cosine=0.10), # below the bar — nobody reads it
_chunk(strong, 0.90, cosine=0.70), # clears the threshold (rank 1)
_chunk(fillers[0], 0.85, cosine=0.30), # ranks 2–5: suggested
_chunk(fillers[1], 0.80, cosine=0.30),
_chunk(fillers[2], 0.75, cosine=0.30),
_chunk(fillers[3], 0.72, cosine=0.30),
_chunk(weak_b, 0.70, cosine=0.20), # rank 6 — related; read by the agent
_chunk(weak_c, 0.65, cosine=0.10), # rank 7 — related; nobody reads it
]
),
)
@@ -815,7 +838,10 @@ def test_agent_read_below_floor_doc_joins_sources(
assert "WEAK_B_READ_BY_AGENT" in tool_msgs[0]["content"]
sources = [(s["source"], s["path"]) for s in done["sources"]]
assert sources == [("docs", "strong.md"), ("docs", "weak-b.md")] # read ⇒ cited, last
# A4: suggested (5) + the read doc (last — it was not suggested).
assert sources[-1] == ("docs", "weak-b.md") # read ⇒ cited, last
assert len(sources) == 6
assert ("docs", "weak-c.md") not in sources # never suggested, never read
related = [(s["source"], s["path"]) for s in done["related"]]
assert related == [("docs", "weak-c.md")] # the read doc is not "nearby"
assert set(sources).isdisjoint(set(related))
@@ -823,3 +849,4 @@ def test_agent_read_below_floor_doc_joins_sources(
(row,) = session.added
assert isinstance(row, QueryLog)
assert "weak-b.md" in row.sources # the full retrieval is recorded (A3)
assert "weak-c.md" in row.sources # … rank 6+ included (suggested + related + read)