phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
@@ -589,3 +589,136 @@ listing answers to reach the file level by default, the lever is the
|
||||
`ls` description / `TOOLS_SECTION` copy (teach that a folder's file
|
||||
lines appear only after drilling into it) — flagged here rather than
|
||||
acted on unilaterally.
|
||||
|
||||
---
|
||||
|
||||
## 10. Phase 118 — summary-seed gate, 2026-09-15 (run 2026-09-16)
|
||||
|
||||
Phase 118 re-worked the grounded-turn seed: the `<documents>` section
|
||||
no longer carries the **full text** of the top-2 cited documents — it
|
||||
carries the **summaries** of the top 5 ranked documents (the
|
||||
"start here if these summaries seem right to you" tier), and a
|
||||
document's full text enters the context only through the capped `read`
|
||||
tool. The fixture battery had to be re-run against the real configured
|
||||
chat model under the re-cut gate semantics (locked A7, owner-confirmed
|
||||
2026-09-15) before the phase could complete — the phase-70/72/94
|
||||
tool-copy gate precedent.
|
||||
|
||||
**Gate semantics changed (locked A7).** The phase gate is now
|
||||
condition **1** (all turns answer) AND condition **2** (zero round-cap
|
||||
hits) AND condition **4** (fixture-mode contract accuracy ≥ 0.90).
|
||||
Condition **3** (≥6/10 turns emit ≥1 tool call) is **REPORTED, not
|
||||
gated** — a summary-only answer is now the *intended* fast path, not
|
||||
tool abandonment. The gate script still gates condition 3 on a full
|
||||
(non-`--turns`) run and exits 1 if it is the only miss; the phase
|
||||
verdict reads the four printed conditions under A7, so a run that
|
||||
misses **only** condition 3 with 1/2/4 green is a **PASS** (recorded
|
||||
as such, below). The full telemetry is recorded regardless.
|
||||
|
||||
**Fixture dump rebuild (a legitimate re-run trigger).** The import
|
||||
pipeline's summary policy changed (phase 118, task 01 — every document
|
||||
gets a stored summary + one embedded `is_summary` chunk, markdown
|
||||
included), so the dump was rebuilt with
|
||||
`uv run python -m scripts.load_test_kb` (the real pipeline). All eight
|
||||
hand-written fixture documents are markdown and each now carries a
|
||||
`doc.summary` + one `is_summary` chunk: **8 documents, 17 chunks**
|
||||
(9 content + 8 summary). The build verified
|
||||
`summary_errors=0`, zero failed folder-summary batches, and an
|
||||
all-GROUNDED retrieval report (the honesty gate is untouched by this
|
||||
phase — grounded/deflected is unchanged). The dump is tracked and
|
||||
committed with the phase.
|
||||
|
||||
The rebuild also surfaced and fixed a latent dump bug: the
|
||||
`_TABLES` column lists in `scripts/load_test_kb.py` predated three
|
||||
migrations, so the snapshot omitted `documents.created_at` /
|
||||
`documents.created_at_manual` (phase 106),
|
||||
`git_sources.ignore_paths` / `git_sources.include_hidden`, and
|
||||
`folder_summaries.manually_edited` (phase 97). Restoring such a dump
|
||||
silently re-defaulted `created_at` to `now()`, so the build's own
|
||||
round-trip checksum verification FAILED on `documents`. The column
|
||||
lists now cover every column of each table, and the round-trip
|
||||
verification passes.
|
||||
|
||||
**The verdict run.** `uv run python -m scripts.agent_realmodel_check
|
||||
--restore --mode fixture` — the configured chat model (`turbo` per
|
||||
`.env`), the refreshed fixture KB, the real grounded path, the full
|
||||
10-question battery (the verdict is always the full battery —
|
||||
`--turns N` is the copy-iteration micro-loop only):
|
||||
|
||||
```
|
||||
$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||||
restore: ok in 0.04s (8 docs, 2 sources)
|
||||
turn 01 | emitted=1 executed=1 cap=no defl=no | 9.49s | List the files in this directory.
|
||||
turn 02 | emitted=4 executed=4 cap=no defl=no | 15.22s | List the documents you have in the …
|
||||
turn 03 | emitted=8 executed=8 cap=no defl=no | 24.16s | List every document you have indexed.
|
||||
turn 04 | emitted=1 executed=1 cap=no defl=no | 12.55s | Open the document …
|
||||
turn 05 | emitted=1 executed=1 cap=no defl=no | 10.45s | Read …
|
||||
turn 06 | emitted=1 executed=1 cap=no defl=no | 7.99s | Open the document …
|
||||
turn 07 | emitted=1 executed=1 cap=no defl=no | 6.63s | Find the exact string "rbm-8842" in …
|
||||
turn 08 | emitted=0 executed=0 cap=no defl=no | 9.92s | Which document has the title "Lab …
|
||||
turn 09 | emitted=1 executed=1 cap=no defl=no | 10.57s | What do you know about the qwen 3.8 …
|
||||
turn 10 | emitted=3 executed=3 cap=no defl=no | 11.00s | List the files in the deployments …
|
||||
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=9 calls 21/21 executed (100%) contract 21/21 (100%) 2026-09-16 (wall 118.1s)
|
||||
```
|
||||
|
||||
**The four conditions (locked A7) and metrics.**
|
||||
|
||||
| condition | gated under A7 | result |
|
||||
|---|---|---|
|
||||
| 1. all turns answer | yes | 10/10 answered — **GREEN** |
|
||||
| 2. zero round-cap hits | yes | caps=0 — **GREEN** |
|
||||
| 3. ≥6/10 turns emit ≥1 tool call | **no (reported)** | 9/10 tool-turns |
|
||||
| 4. contract accuracy ≥ 0.90 | yes | 21/21 (100%) — **GREEN** |
|
||||
| executed / emitted (reported) | no | 21/21 (100%) |
|
||||
|
||||
Contract line: **contract 21/21 (100%)**. Wall time: **118.1 s**.
|
||||
Model: **turbo** (the configured chat model).
|
||||
|
||||
**Per-turn reading.** The three designed read turns (04/05/06) each
|
||||
emitted exactly one contract-correct `read` in the combined
|
||||
`source/path` form — and each **executed** (the seeded document is now
|
||||
a *summary*, so `read` adds its full text rather than being
|
||||
dedupe-refused: 100 % executed, not the ~58–73 % the full-text seed
|
||||
produced). The discipline turns behave exactly as the new contract
|
||||
wants: turn 08 (title lookup, target seeded as a summary) answered
|
||||
straight from the `<documents>` context with **zero** tool calls, and
|
||||
turn 09 (the qwen launch arguments) answered from the seeded summary
|
||||
context. The listing turns drill the tree as under phase 94 (turn 03:
|
||||
top level + both sources + the leaf folders in 8 calls). Zero cap
|
||||
hits, zero contract violations, 10/10 answered.
|
||||
|
||||
**Baseline comparison.** Against the phase-94 `turbo` baseline
|
||||
(98.7 s wall, 100 % contract, 100 % executed on 19 calls): contract
|
||||
and executed are unchanged at 100 %, caps remain 0, and the wall time
|
||||
is 118.1 s (+19.7 %) — inside the ~20 % band the phase-94 gate used
|
||||
as its slowdown tripwire, and within the normal endpoint-load
|
||||
variance of the 97–135 s `turbo` range recorded in §3. The gate is
|
||||
met on the conditions that gate (1/2/4 all GREEN) and condition 3 is
|
||||
reported (9/10).
|
||||
|
||||
**`read` copy — final shape (no copy iteration was needed).**
|
||||
Condition 4 did not degrade and condition 2 was never hit, so no
|
||||
`--turns` micro-loop iteration was required. The final copy is the
|
||||
task-04 rewrite, byte-pinned in `tests/unit/test_prompt_lock.py` /
|
||||
`test_agent.py`: the `read` description in `AGENT_TOOLS`
|
||||
(`app/rag/agent.py`) teaches that `<documents>` shows the *summaries*
|
||||
of the top-ranked documents (their full texts are NOT in the prompt
|
||||
yet) and that `read` adds one of them — or any other document — by
|
||||
its combined `source/path`, and must not re-read an already-read
|
||||
document (that refusal's copy stays accurate: "the full text is
|
||||
already in your prompt"); the `read` clause of `TOOLS_SECTION`
|
||||
(`app/rag/prompts.py`) mirrors it ("The `<documents>` section holds
|
||||
SUMMARIES — the full text of a suggested document is not in your
|
||||
prompt until you `read` it"). `classify_call` needed **no**
|
||||
re-baselining: a `read` of a suggested document is a well-formed call
|
||||
targeting a resolvable document — contract-correct under both the old
|
||||
and the new app semantics (its docstring's `ALREADY_IN_CONTEXT` note
|
||||
already excludes dedupe re-reads from the metric).
|
||||
|
||||
**Conclusion.** The phase-118 summary-seed gate is **PASS**: conditions
|
||||
1, 2, and 4 GREEN (condition 3 reported at 9/10, not gated), contract
|
||||
accuracy 100 %, zero cap hits, the full battery all-answered, wall
|
||||
within the ~20 % band. The summary-seed behavior is proven against the
|
||||
real configured chat model — full text enters the context only through
|
||||
the capped `read` tool, and a summary-only answer is the intended fast
|
||||
path.
|
||||
|
||||
Reference in New Issue
Block a user