phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
+169
-66
@@ -221,50 +221,71 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
"'homelab/active'). Omit it to list every source."
|
||||
)
|
||||
read = by_name["read"]["function"]
|
||||
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
|
||||
# the do-not-read rule is FRONT-LOADED — the controlled gate's
|
||||
# telemetry showed the `lite` model obeying the user's "open it /
|
||||
# read it" and reading seed-context documents the <documents>
|
||||
# section already carries (every refusal of a 12-call run was
|
||||
# ALREADY_IN_CONTEXT); the rule now leads the description instead
|
||||
# of sitting mid-paragraph, and the tool is framed as "only for
|
||||
# documents NOT already in <documents>". Phase 95 (task 01): the
|
||||
# read-truncation sentence is inserted before the one-call-at-a-
|
||||
# time discipline clause (the discipline rule stays last, as in the
|
||||
# other two tools) — a capped read carries the TRUNCATED notice and
|
||||
# the `grep` follow-up (the pinned copy).
|
||||
# Phase 118 (task 04, A6): the read description is rewritten for
|
||||
# the summary-seed mode — the <documents> section shows the
|
||||
# top-ranked documents' SUMMARIES (their full texts are NOT in
|
||||
# the prompt yet); read adds one of them (or any other document)
|
||||
# to the context; only an already-read document is refused. The
|
||||
# combined-identity handoff to the `ls` output / the <documents>
|
||||
# blocks stays; the truncation-notice paragraph (phase 95) and
|
||||
# the one-call-at-a-time discipline clause (task 05) survive the
|
||||
# rewrite byte-identical.
|
||||
assert read["description"] == (
|
||||
"Do not call this tool for a document already shown in "
|
||||
"the <documents> section, even when the user asks you to "
|
||||
"open or read it — its full text is already in your "
|
||||
"prompt; answer directly from it. Use it only to add a "
|
||||
"document NOT already in <documents> to your context, "
|
||||
"by its combined `source/path` string. Very large "
|
||||
"documents are truncated: you receive the first part "
|
||||
"plus a TRUNCATED notice naming how many more characters "
|
||||
"exist — the notice is authoritative, the document did "
|
||||
"NOT end where it stopped. Follow it and use `grep` "
|
||||
"(pattern) to locate the rest — it searches the whole "
|
||||
"document. Call one tool at a time — wait for this "
|
||||
"result before your next call."
|
||||
"The <documents> section shows the SUMMARIES of the "
|
||||
"top-ranked documents — their full texts are NOT in "
|
||||
"your prompt yet. Use this tool to add one of them (or "
|
||||
"any other document) to your context, by its combined "
|
||||
"`source/path` string, exactly as shown in the `ls` "
|
||||
"output or the <documents> blocks. Do not re-read a "
|
||||
"document you have already read — its full text is "
|
||||
"already in your prompt. Very large documents are "
|
||||
"truncated: you receive the first part plus a TRUNCATED "
|
||||
"notice naming how many more characters exist — the "
|
||||
"notice is authoritative, the document did NOT end "
|
||||
"where it stopped. Follow it and use `grep` (pattern) "
|
||||
"to locate the rest — it searches the whole document. "
|
||||
"Call one tool at a time — wait for this result before "
|
||||
"your next call."
|
||||
)
|
||||
# The byte-preserved contracts inside the rewrite, pinned as
|
||||
# substrings: the combined source/path identity, the
|
||||
# truncation-notice paragraph, the one-call-at-a-time
|
||||
# discipline sentence.
|
||||
assert (
|
||||
"by its combined `source/path` string, exactly as shown in "
|
||||
"the `ls` output or the <documents> blocks"
|
||||
) in read["description"]
|
||||
assert (
|
||||
"Very large documents are truncated: you receive the first "
|
||||
"part plus a TRUNCATED notice naming how many more "
|
||||
"characters exist — the notice is authoritative, the "
|
||||
"document did NOT end where it stopped. Follow it and use "
|
||||
"`grep` (pattern) to locate the rest — it searches the "
|
||||
"whole document."
|
||||
) in read["description"]
|
||||
assert (
|
||||
"Call one tool at a time — wait for this result before "
|
||||
"your next call."
|
||||
) in read["description"]
|
||||
read_params = read["parameters"]
|
||||
assert read_params["type"] == "object"
|
||||
assert read_params["required"] == ["path"]
|
||||
assert set(read_params["properties"]) == {"path"}
|
||||
assert read_params["properties"]["path"]["type"] == "string"
|
||||
# The combined source/path string is the canonical document identity
|
||||
# (phase 70) — the description pins it with a worked example. Phase
|
||||
# 72 (task 02): the bare-path contract is stated up front; task 05
|
||||
# (live gate iteration 1): the do-not-re-read clause (the dedupe
|
||||
# refusal's prevention at the prompt).
|
||||
# (phase 70) — the description pins it with a worked example; the
|
||||
# bare-path contract is stated up front (phase 72, task 02). Phase
|
||||
# 118 (task 04, A6): the trailing clause is the do-not-RE-READ
|
||||
# teaching — the seeds are summary blocks, not full text, so a
|
||||
# first read of a suggested document succeeds and only an
|
||||
# already-read document is refused.
|
||||
assert read_params["properties"]["path"]["description"] == (
|
||||
"The document to add to your context, as the combined "
|
||||
"`source/path` string exactly as shown in the `ls` output (e.g. "
|
||||
"'homelab/active/container_caddy/caddy.md'). A bare document "
|
||||
"path (without the source name) will not resolve. Only pass a "
|
||||
"document NOT already shown in the <documents> section — it is "
|
||||
"already in your context; do not re-read it."
|
||||
"path (without the source name) will not resolve. Do not "
|
||||
"re-read a document you have already read — its full "
|
||||
"text is already in your prompt."
|
||||
)
|
||||
grep = by_name["grep"]["function"]
|
||||
# Task 05 (live gate iterations 2-6, refined in the 2026-09-03
|
||||
@@ -1455,15 +1476,17 @@ def test_read_bare_filename_without_slash_keeps_no_db_refusal(
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
def test_read_bare_path_of_suggested_doc_gets_suggestion_then_succeeds(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Dedupe precedence: the in-context dedupe fires on the SPLIT pair
|
||||
of the argument — the bare path of an in-context document
|
||||
(``read('app/rag/importer.py')`` with ``sample/app/rag/importer.py``
|
||||
seeded) is NOT that pair, so it is not a dedupe: it gets the
|
||||
suggestion line naming the combined identity, and the model's next,
|
||||
correctly-formed call is then deduped as ALREADY_IN_CONTEXT."""
|
||||
"""Dedupe precedence (phase 118): the in-context dedupe fires on
|
||||
the SPLIT pair of a document already READ — the bare path of a
|
||||
SUGGESTED (seeded) document (``read('app/rag/importer.py')`` with
|
||||
``sample/app/rag/importer.py`` seeded) is NOT that pair, so it is
|
||||
not a dedupe: it gets the suggestion line naming the combined
|
||||
identity, and the model's next, correctly-formed call ADDS the
|
||||
suggested document's full text (A6: the seeds are summary blocks,
|
||||
not full text — a first read of a suggested document succeeds)."""
|
||||
seed = [_doc("sample", "app/rag/importer.py", "Importer", "IMPORTER")]
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
@@ -1482,8 +1505,9 @@ def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
],
|
||||
[
|
||||
# Round 2: the corrected call (the suggested combined
|
||||
# identity) — the seed document is already in context, so it
|
||||
# dedupes.
|
||||
# identity) — the suggested document's full text is NOT in
|
||||
# the prompt (only its summary is), so this read succeeds
|
||||
# and adds the full content.
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
@@ -1493,12 +1517,18 @@ def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # both refused
|
||||
# Round 1 refused (the teaching), round 2 executed (A6).
|
||||
assert holder.read_docs == [seed[0]]
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'app/rag/importer.py' — "
|
||||
"did you mean 'sample/app/rag/importer.py'?"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert llm.requests[2][0][5]["content"] == (
|
||||
"Document sample/app/rag/importer.py:\n"
|
||||
"date: 2024-06-15\n"
|
||||
"IMPORTER"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
@@ -1529,16 +1559,23 @@ def test_read_missing_arguments_refused(
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_reading_a_seed_doc_is_already_in_context(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The combined identity of a seeded document: its split pair is in
|
||||
the known set → ALREADY_IN_CONTEXT with no DB lookup (the dedupe
|
||||
check precedes the resolve)."""
|
||||
def test_reading_a_suggested_seed_doc_adds_its_full_text(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 118 (A6): a FIRST read of a suggested (seeded) document
|
||||
SUCCEEDS — the seed is a summary block in the prompt, not the full
|
||||
text, so it falls out of the dedupe set (``holder.read_docs``
|
||||
only): the read goes through the existing path unchanged — the
|
||||
full-content result (header + date line), appended to
|
||||
``holder.read_docs``, counted in ``holder.tool_calls``."""
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("find_document must not be called for a seeded doc")
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
monkeypatch.setattr(
|
||||
agent,
|
||||
"find_document",
|
||||
lambda db, source, path: seed[0]
|
||||
if (source, path) == ("Homelab", "kubernetes.md")
|
||||
else None,
|
||||
)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
@@ -1551,11 +1588,58 @@ def test_reading_a_seed_doc_is_already_in_context(monkeypatch: pytest.MonkeyPatc
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_docs == [seed[0]]
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document Homelab/kubernetes.md:\ndate: 2024-06-15\nK8S-CONTENT"
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_re_reading_a_suggested_doc_is_deduped(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 118 (A6): once the suggested document's full text has been
|
||||
read into context, a SECOND read of the same combined identity is
|
||||
refused with the byte-identical ``ALREADY_IN_CONTEXT`` line — the
|
||||
dedupe set is ``holder.read_docs`` only, so the refusal fires with
|
||||
NO DB lookup (the dedupe check precedes the resolve) and the
|
||||
counters stay untouched."""
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
lookups: list[tuple[str, str]] = []
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
lookups.append((source, path))
|
||||
return seed[0] if (source, path) == ("Homelab", "kubernetes.md") else None
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "Homelab/kubernetes.md"},
|
||||
)
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
arguments={"path": "Homelab/kubernetes.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [seed[0]] # appended exactly once
|
||||
assert holder.tool_calls == 1 # the re-read counts nothing
|
||||
assert lookups == [("Homelab", "kubernetes.md")] # the re-read deduped pre-lookup
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document Homelab/kubernetes.md:\ndate: 2024-06-15\nK8S-CONTENT"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
# Rejected → the tools are still offered on the next request (the
|
||||
# round cap is the only bound).
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
@@ -1718,28 +1802,47 @@ def test_run_agent_short_read_yields_no_tool_result_piece(
|
||||
assert isinstance(out[1], StreamPiece)
|
||||
|
||||
|
||||
def test_read_truncation_does_not_touch_refusal_paths(
|
||||
def test_read_truncation_paths_with_suggested_seed(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95: the read refusal paths are untouched by the cap — a SEED
|
||||
document that is over the cap is still refused with
|
||||
``ALREADY_IN_CONTEXT`` (not truncated, nothing recorded, nothing
|
||||
counted), and an unknown path is still the no-document refusal (no
|
||||
"""Phase 118 (A6) re-target of the phase-95 pins ("seed" →
|
||||
"suggested"): a SUGGESTED (seeded) document that is over the cap is
|
||||
read through the ordinary cap path — cut at the cap + the shared
|
||||
marker + the pinned notice, the truncation recorded on the holder,
|
||||
and the call still successful (the seed is a summary, not full
|
||||
text) — and an unknown path is still the no-document refusal (no
|
||||
content is read, so no truncation either)."""
|
||||
big = "B" * 5000 # far over the tiny cap below
|
||||
seed = _doc("S", "seed.md", "Seed", big)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [])
|
||||
# (a) Reading the (over-cap) seed doc → ALREADY_IN_CONTEXT (refusal).
|
||||
monkeypatch.setattr(
|
||||
agent,
|
||||
"find_document",
|
||||
lambda db, source, path: seed
|
||||
if (source, path) == ("S", "seed.md")
|
||||
else None,
|
||||
)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [seed])
|
||||
# (a) Reading the (over-cap) SUGGESTED doc → the capped read: cut at
|
||||
# the cap, marker + pinned notice, holder entry, still counted.
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/seed.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=100), seed_docs=[seed]))
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_truncations == []
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/seed.md:\n"
|
||||
"date: 2024-06-15\n"
|
||||
+ big[:100]
|
||||
+ "\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=100, total=5000)
|
||||
)
|
||||
# (argument, cap, total) — the raw argument, the cap kept, the
|
||||
# true length.
|
||||
assert holder.read_truncations == [("S/seed.md", 100, 5000)]
|
||||
assert holder.tool_calls == 1 and holder.read_docs == [seed]
|
||||
# (b) An unknown path → the no-document refusal (argument echoed),
|
||||
# even though a big doc could have truncated — no content is read.
|
||||
holder2 = AgentHolder()
|
||||
|
||||
Reference in New Issue
Block a user