chore(agent): phase roadmap from TODO.md — phase 118 summary seed context (top-5 summary seeding, all-doc summaries)
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# Phase 118 — Summary seed context: top-5 "start here" suggestions (TODO L3)
|
||||
|
||||
**Source:** `TODO.md` L3 — "I want to change the way the initial chat context works. Right now we feed two entire documents into context before the LLM responds. That was a good idea before we had complex tools and document summaries, but now we have faster ways for the LLM to identify what information it needs without reading 2 files. Moving forward, after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant because the similarity search had a bad day."
|
||||
**Story:** n/a (owner-directed RAG behavior change; extends the phase-30 summaries, the phase-37/70/94 agent tools, and the phase-113 tiering assets).
|
||||
**Context:** A grounded (HIGH) turn currently seeds `build_high_prompt` with the **full text** of the top-2 cited documents (`select_documents_tiered`'s cited tier — floor `BOR_SOURCE_USEFULNESS_FLOOR` 0.35, ceiling `BOR_TOP_N_DOCS` 2, `app/api/chat.py:plan_turn`), and `run_agent` receives them as `seed_docs` whose `read` is refused (`ALREADY_IN_CONTEXT` — the full text is already in the prompt). Summaries exist on `Document.summary` (phase-30 lite digest — currently **non-markdown only**; NULL for markdown docs, pre-30 rows and fail-soft misses) and are embedded as one `is_summary` chunk (position −1) per document. LOCKED A7 (full text of the top-2, never truncated, on the retrieval path) is **re-revised by the owner directive in this TODO**: full text enters the context only through the capped `read` tool (locked A6 below).
|
||||
|
||||
## Objective
|
||||
Grounded turns seed the top 5 related documents as **summary** blocks — a "start here if these summaries seem right to you" starting point — and the LLM extends its context by `read`-ing only what it needs; every document (markdown included) has a stored summary (generated for all new/changed docs at import, backfilled on the next sync for existing NULL rows). Answers get faster and never start by ingesting massive possibly-irrelevant full texts.
|
||||
|
||||
## Dependencies
|
||||
- `117_tool_call_compact` (complete) — the last completed phase; this phase rewrites the `read`-tool copy the battery gate pins, so the gate (task 07) runs against its final state.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Suggested tier (task 02):** new pure `select_suggested(chunks, n=None)` in `app/rag/retriever.py` — the top-N DISTINCT parent documents in fused rank order (the same stable score-descending walk as `select_documents_tiered` — a document's rank is fixed by its FIRST seen chunk), **no cosine floor** (suggestions are opt-in starting points, not citations — the floor stays for the `done`-event related tier only). `n` defaults to the new setting `suggested_docs` (`BOR_SUGGESTED_DOCS`, default **5**, validator `>= 1`).
|
||||
- **Suggestion blocks (task 03):** the HIGH prompt keeps the `<documents>` section tag and the per-block `<document …>` markup (the E2E mock keys off both), but each block's BODY becomes the document's **summary**: the identity attributes (`source`/`path`/`title`/`date` — the phase-106 D5 `date` stays, always present) + `doc.summary` (stripped). Defensive fallback ONLY when `doc.summary` is None/whitespace (a fail-soft miss — locked A5): the first `suggestion_preview_chars` (`BOR_SUGGESTION_PREVIEW_CHARS`, default **400**) content characters + the shared `TRUNCATION_MARKER` — no LLM call at chat time. A new pinned intro line (a module constant in `app/rag/prompts.py`) precedes the blocks inside the section: the start-here framing — the blocks are summaries of the top-ranked documents; if one seems right, `read` that document (combined `source/path`) to add its full content; answer from what you have and what you read; cite the document(s) you used by path.
|
||||
- **Agent `read` (task 04):** with summary seeds, `read` on a suggested document **adds its full text** — the existing mechanics are all unchanged (the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `ToolResultPiece`/SSE `tool_result` frame, the phase-106 D5 `date:` line, `holder.read_docs` append, `tool_calls` increment). The dedupe set in `_execute_tool`'s read branch becomes **`holder.read_docs` only** (the seeds drop out — a suggested doc is a summary, not full text), so `ALREADY_IN_CONTEXT` fires only for a document ALREADY READ into full-text context (its copy stays accurate: "the full text is already in your prompt"). The `read` tool description in `AGENT_TOOLS` (`app/rag/agent.py`) and the `read` clause of `TOOLS_SECTION` (`app/rag/prompts.py`) are rewritten for the summary-seed mode — SINGLE mode (no kill switch, locked A1): the `<documents>` section shows summaries, `read` adds a document's full text, and re-reading an already-read document is refused. The `ls`/`grep` clauses and the discipline rules (one call per reply, never repeat) stay byte-identical. `tests/unit/test_prompt_lock.py` byte-pins `TOOLS_SECTION` (sha256 + length + prefix/suffix): re-cut its anchors in the same change; `PERSONA` and the LOW `DEFLECT_MODE` body MUST stay byte-identical (the lock pins those too — the E2E mock keys on the markers' presence).
|
||||
- **Turn wiring (task 05):** `plan_turn` computes BOTH tiers once, for BOTH branches: `suggested = select_suggested(chunks)` and `related = select_related(chunks, excluded={suggested ids}, cap=settings.related_max_docs)` (new pure helper in `app/rag/retriever.py` — the same rank walk, skipping the excluded ids, no floor; with a 5-doc suggestion the related tier is rank order 6..n, the phase-113 UI "nearby docs" row). The HIGH branch: `build_high_prompt(suggested, …)` and `run_agent(..., seed_docs=suggested)`; the LOW branch: prompt BYTE-IDENTICAL (weak-hit titles only, `DEFLECT_MODE`, no tools, chips — A8 gate untouched) but the TurnPlan still carries suggested + related for the durable record. `TurnPlan.docs` is RENAMED `suggested_docs` (same dataclass position); `summary_hits` is redefined as the hit chunks with `is_summary` whose parent document is in the SUGGESTED set. Citations: `done.sources` = suggested + agent-read, deduped by `(source, path)` order-preserving (locked A4 — a grounded turn always shows chips; the deflected turn keeps `[]`). Durable record (LOCKED A3 — the log records retrieval, not citations): `query_log.sources` + the per-turn log line = suggested + related + read, deduped. The per-turn log line gains `suggested=N` after `summary_hits=N` (PLAN §9 extension). `source_usefulness_floor` stops being a seeding input (it may stay for other consumers — the executor checks; the SETTING itself is never removed).
|
||||
- **Summaries for every document (task 01):** the importer's summary step covers **every** new/changed A9 document — the markdown exclusion (`_import_one_file`'s `if suffix in (".md", ".markdown"): return`) is removed. **Backfill:** on the UNCHANGED branch (same `content_hash`), a document whose `summary is None` gets the same best-effort `_store_summary` pass — summary text stored on `doc.summary` + one new embedded `is_summary` chunk — with a `backfill=True` flag that counts success in a NEW `ImportSummary.summary_backfilled` counter (instead of `summaries`). The backfill runs BEFORE the `created_at_manual` early-return and the date refresh (the manual flag protects the DATE only, phase 106 D1 — a manually-dated doc still gets its summary), is still fail-soft (`LLMError`/`EmbeddingError` → session rollback + `summary_errors += 1`, the doc row untouched, the date refresh still runs), and NEVER counts `added`/`updated`/`pruned` (so no `sources_meta` bump, no KB-overview/folder-summary regeneration — the content is unchanged, locked A2). Non-NULL summaries (owner-edited via phase 57 included) are never touched. The import log line (PLAN §9) gains `summary_backfilled=N` after `summary_errors=N`.
|
||||
- **Old-contract retirement (task 06):** `tests/e2e/test_whole_document_context.py` (the phase-24 full-text sentinel E2E) is DELETED — its contract no longer exists (locked A1/A6). Every test that pins the retired behavior is updated in place to pin the new one: markers to audit across `tests/unit/`, `tests/integration/`, `tests/e2e/` — the `ALREADY_IN_CONTEXT` seed-read flows (`test_agent.py`, `test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`), the `<documents>` full-text pins (`test_prompts.py`, `test_prompts_dates.py`, `test_chat_gate.py`), the markdown-summary pin (`test_importer.py::test_markdown_file_never_gets_summary` — inverted), the per-turn log-line pins (`test_chat_gate.py` — `summary_hits=`). A pin of UNCHANGED behavior stays untouched.
|
||||
- **Copy gate (task 07):** the `read`-tool copy is the lever the gate telemetry proved matters (the phase-70/72/94 precedent) — the real-model fixture battery re-runs before the phase completes and its verdict is recorded in `TOOL_CALLING_TESTING.md` (gate semantics: locked A7 below).
|
||||
|
||||
## Tasks
|
||||
1. `01_all_doc_summaries.md` — importer: every document gets a summary (markdown included) + the NULL-summary backfill on the unchanged path.
|
||||
2. `02_suggested_tier_selection.md` — retriever: `select_suggested` (top-N, no floor) + `BOR_SUGGESTED_DOCS`.
|
||||
3. `03_summary_document_blocks.md` — prompts: the "start here" suggestion blocks in the HIGH prompt (+ the preview fallback).
|
||||
4. `04_read_suggested_docs.md` — agent: `read` on a suggested doc adds its full text; the `read` copy rewritten (both surfaces); the prompt-lock re-cut.
|
||||
5. `05_chat_turn_wiring.md` — `plan_turn`: suggested seeding, `select_related`, citations, the durable record, `suggested=N`.
|
||||
6. `06_e2e_summary_context.md` — new Playwright E2E; the phase-24 E2E retired; the old pins updated.
|
||||
7. `07_fixture_battery_gate.md` — the real-model fixture battery re-run + the verdict recorded.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_importer.py` (markdown summarized; backfill success/failure; the manual-date case; the log-line counter), `tests/unit/test_retriever.py` (`select_suggested` / `select_related` rank order, cap, no floor, dedupe), `tests/unit/test_prompts.py` + `test_prompts_dates.py` (summary blocks, the `date` attribute, the preview fallback, the intro line; the LOW prompt byte-identical), `tests/unit/test_prompt_lock.py` (the re-cut `TOOLS_SECTION` anchors; `PERSONA` + LOW body untouched), `tests/unit/test_agent.py` (read-of-suggested success + re-read refusal + cap path), `tests/unit/test_chat_gate.py` (HIGH seeds 5 summaries — no full text in the prompt; related = rank 6+; both branches' durable record; `suggested=N`), `tests/unit/test_config.py` (the two new settings).
|
||||
- E2E: `tests/e2e/test_summary_seed_context.py` (new — run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`).
|
||||
- Regression: the remaining E2E suites green (the tool-flow suites re-targeted in task 06), and the battery gate (task 07).
|
||||
- Coverage: **>90%** on `app/` (the validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A grounded turn seeds the top-5 suggested documents as SUMMARY blocks (no full text in the prompt); the LLM `read`s what it needs and answers faster.
|
||||
- [ ] Every imported document (markdown included) has a `summary`; an existing NULL-summary doc backfills on the next sync (no content re-embed, no `sources_meta` bump, fail-soft).
|
||||
- [ ] `read` on a suggested doc adds its full text (cap/truncation/date line unchanged); a re-read of an already-read doc is refused.
|
||||
- [ ] `done.sources` = suggested + read (deduped); `query_log` + the log line record suggested + related + read; the log line carries `suggested=N`.
|
||||
- [ ] The deflection path and every SSE frame shape are byte-identical; `PERSONA` + the LOW body stay byte-locked.
|
||||
- [ ] `uv run pytest` green; coverage >90%; the new E2E green in isolation; the fixture battery gate PASS (task 07); `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — no kill switch (owner-confirmed 2026-09-15, roadmap confirmation):** the summary-seed context IS the behavior; the legacy full-text top-2 seeding is gone and the phase-24 E2E is retired (not re-scoped).
|
||||
- **A2 — every document gets a summary, no exceptions, including markdown (owner-confirmed 2026-09-15):** generated for all new/changed docs at import; existing NULL rows backfill on the next sync as a summary-only pass (no content re-chunk/re-embed, no `sources_meta` bump); owner-edited (non-NULL) summaries are never touched.
|
||||
- **A3 — the suggestion set = the top 5 distinct documents in fused rank order, NO cosine floor** (`BOR_SUGGESTED_DOCS` default 5); the floor stays for the `done`-event related tier only (owner-confirmed 2026-09-15).
|
||||
- **A4 — `done.sources` (the citation surface) = suggested + agent-read, deduped** — a grounded turn always shows chips (owner-confirmed 2026-09-15).
|
||||
- **A5 — a NULL summary at prompt time (a fail-soft miss) falls back to a 400-char content preview** (`BOR_SUGGESTION_PREVIEW_CHARS`) — deterministic, no LLM call at chat time (owner-confirmed 2026-09-15).
|
||||
- **A6 — A7 re-revised (owner directive, TODO L3, 2026-09-15):** the retrieval path never seeds full texts; full text enters the context only through the capped `read` tool. `PLAN.md` is untouched (Protocol B) — the owner folds it in at the next redo.
|
||||
- **A7 — the battery-gate semantics under summary seeding (owner-confirmed 2026-09-15):** conditions 1 (every turn answers) and 2 (zero round-cap hits) still gate; condition 4 (fixture-mode contract accuracy ≥ 0.90) still gates; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is now the intended fast path, not tool abandonment. The full telemetry is recorded in `TOOL_CALLING_TESTING.md` regardless.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): seed grounded turns with top-5 document summaries instead of full texts — every document summarized, read extends the context"
|
||||
```
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 01 — Summaries for every document + the NULL-summary backfill
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…now we have faster ways for the LLM to identify what information it needs… with their summaries loaded into context rather than their entire content…" (locked A2: every document gets a summary, no exceptions, including markdown.)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Every A9 document has a stored lite summary: the markdown exclusion is dropped (all new/changed docs get one), and an existing unchanged doc whose `summary` is NULL gets a summary-only backfill on the next sync.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py` — `_import_one_file` (~L539–544): delete the markdown early-return (`if full_path.suffix.lower() in (".md", ".markdown"): return`) so EVERY new/changed document falls through to `_store_summary`. Update the adjacent comment (phase 30's "markdown is already natural language" rationale is retired — locked A2, 2026-09-15).
|
||||
2. `app/rag/importer.py` — `_store_summary` (~L548–592): add a `backfill: bool = False` keyword; on success count `summary.summary_backfilled += 1` when `backfill` else `summary.summaries += 1` (the two counters are disjoint — the log line keeps both). Everything else (fail-soft rollback, the `is_summary` chunk replacement, the embed) is unchanged.
|
||||
3. `app/rag/importer.py` — `_import_one_file`'s UNCHANGED branch (~L442–464): after `summary.unchanged += 1` and its log line, BEFORE the `created_at_manual` early-return and the date refresh, add: `if doc.summary is None: await _store_summary(session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary, backfill=True)`. Order matters: a backfill failure rolls back its own session work (nothing else is pending), and the date refresh still runs afterwards. Strict `is None` check — an empty-string summary is owner-set (phase 57) and is never overwritten. A backfill NEVER touches `content_hash`, the content chunks, `created_at` semantics, or the added/updated/pruned counters — so no `sources_meta` bump, no KB-overview/folder-summary regeneration (locked A2).
|
||||
4. `app/rag/importer.py` — `ImportSummary` (~L95–117): new `summary_backfilled: int = 0` field (docstring: unchanged docs whose NULL summary was backfilled — one `is_summary` chunk per success); update the `summaries`/`summary_errors` docstrings ("Non-markdown files" → "Files" — every A9 format now, locked A2); `log()` gains `summary_backfilled=%d` placed after `summary_errors=%d` (PLAN §9 line extension).
|
||||
5. `app/rag/summarizer.py` — NO copy change: `SUMMARY_INSTRUCTION` is already format-agnostic and the `SUMMARY_MODE` marker is the E2E mock's key (byte-stable).
|
||||
6. ASSUMPTION: the backfill is NOT a KB change (locked A2) — no `sources_meta` bump even though a prompt-facing summary appears: the doc content is unchanged, and saved-chat staleness tracks content generations (the phase-53 invariant).
|
||||
7. ASSUMPTION: `created_at_manual` rows still get the backfill — the manual flag protects the DATE only (phase 106 D1), not the summary.
|
||||
|
||||
## Testing & Quality
|
||||
- `tests/unit/test_importer.py` — update/extend:
|
||||
- `test_markdown_file_never_gets_summary` (~L574) → INVERT: a markdown file gets a stored summary + one embedded `is_summary` chunk (rename, e.g. `test_markdown_file_gets_stored_summary`).
|
||||
- new: unchanged doc with NULL summary → backfilled (summary stored, one `is_summary` chunk with embedding, `unchanged` counter, `summary_backfilled=1`, `summaries` untouched, `added/updated/pruned` all 0).
|
||||
- new: unchanged doc WITH a stored summary → no summary LLM call at all (the fake `chat` is never invoked) and no new chunk.
|
||||
- new: unchanged doc with `created_at_manual=True` and NULL summary → summary backfilled, `created_at` byte-untouched.
|
||||
- new: backfill failure (fake `chat` raises `LLMError`) → fail-soft: `summary_errors=1`, `summary_backfilled=0`, the doc row + date refresh unaffected (the date still refreshes when it differs).
|
||||
- `test_import_summary_log_line_includes_summary_counters` (~L719) → the log line carries `summary_backfilled=N`.
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A sync of a KB containing markdown docs stores a summary on every doc (mock LLM `SUMMARY_MODE` path).
|
||||
- [ ] A second sync (all unchanged) backfills exactly the NULL-summary docs; the third sync makes no summary calls at all.
|
||||
- [ ] `uv run pytest tests/unit/test_importer.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 02 — `select_suggested`: the top-N, no-floor suggestion tier
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
A pure `select_suggested(chunks, n)` in the retriever returns the top-N distinct parent documents in fused rank order, with NO cosine floor (locked A3), plus the `BOR_SUGGESTED_DOCS` setting (default 5).
|
||||
|
||||
## Work
|
||||
1. `app/rag/retriever.py` — new pure function `select_suggested(chunks: Sequence[RetrievedChunk], n: int | None = None) -> list[Document]`: walk `chunks` in the SAME stable score-descending order as `select_documents_tiered` (a document's rank is fixed by its FIRST seen chunk; dedupe by `document.id`), return at most *n* documents — **no floor** (a lexical-only hit with cosine 0.0 is suggested when ranked, unlike the phase-113 cited tier). `n` defaults to `get_settings().suggested_docs`. Docstring: the phase-118 "start here" suggestion tier — opt-in starting points, not citations (locked A3); the returned rows carry the full document content, byte-identical (the content is what `read` serves later — never truncated, A6).
|
||||
2. `app/config.py` — new `suggested_docs: int = 5` (env `BOR_SUGGESTED_DOCS`), validator `>= 1` (the `related_max_docs` non-negative style, `app/config.py` ~L381–387), docstring citing the phase-118 "start here" seeding; `.env.example` entry with the same comment (near the `BOR_TOP_N_DOCS` / `BOR_RELATED_MAX_DOCS` block).
|
||||
3. `tests/unit/test_config.py` — the setting default + the validator (0 and negative rejected, 1 accepted).
|
||||
4. `tests/unit/test_retriever.py` — `select_suggested`:
|
||||
- rank order = best fused score per document, first-seen-chunk rank (the same walk — a doc whose best chunk appears later in the list still ranks by first-seen position, matching `select_documents_tiered`'s order semantics);
|
||||
- dedupe (multiple chunks of one doc → one row);
|
||||
- the cap (`n` and the settings default);
|
||||
- NO floor: a lexical-only 0.0-cosine document is included when ranked (contrast pin against `select_documents_tiered`'s floored cited tier on the same input);
|
||||
- deterministic tie-break inherited from the fused order.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `select_suggested` returns the top-5 distinct docs in fused rank order for a typical fused list; the floor never filters.
|
||||
- [ ] `BOR_SUGGESTED_DOCS` is env-tunable (default 5) and validated.
|
||||
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 03 — The "start here" suggestion blocks in the HIGH prompt
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The HIGH prompt's `<documents>` section seeds each suggested document as an identity + SUMMARY block (never the full content), with a pinned intro line carrying the start-here framing; a NULL summary (fail-soft miss) falls back to a 400-char content preview (locked A5).
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `build_high_prompt`: keep the `<documents>` section tag and the per-block `<document source=… path=… title=… date=…>` markup BYTE-STABLE (the E2E mock's `_document_block` parser and the `read`-tool copy key off both) — only the block BODY changes: `doc.summary.strip()` instead of `doc.content`. The `date` attribute stays (phase 106 D5, always present).
|
||||
2. `app/rag/prompts.py` — the preview fallback (locked A5): when `doc.summary` is None or whitespace-only, the body is `doc.content[:limit]` + the shared `TRUNCATION_MARKER` on its own line — `limit` from `get_settings().suggestion_preview_chars`, read ONLY on that fallback path (the house pattern: `build_steering_section`/`build_kb_section` read settings inside the function; `build_high_prompt`'s signature is unchanged, every existing caller works, and a prompt built from summary-bearing docs makes no settings call for the cap). New setting in `app/config.py`: `suggestion_preview_chars: int = 400` (env `BOR_SUGGESTION_PREVIEW_CHARS`, validator `> 0`) + `.env.example` entry.
|
||||
3. `app/rag/prompts.py` — new module constant `SUGGEST_INTRO` (pinned copy, the phase-15 `_STEERING_INTRO`/phase-31 `_KB_INTRO` precedent) rendered inside the `<documents>` section, BEFORE the first block, when at least one block is present. Copy (adjust wording freely, keep the three contracts below, pin the final text in `test_prompt_lock.py` or `test_prompts.py`): the blocks are SUMMARIES of the top-ranked documents for the question — start here if one seems right; call `read` with the document's combined `source/path` to add its FULL content before answering specifics; a document's full text is NOT in the prompt until you read it; cite the document(s) you used by path. The empty-body fallback line (`"(no documents matched — do not invent specifics)"`) stays.
|
||||
4. `app/rag/prompts.py` — docstrings: the module docstring's HIGH-mode description ("full top-document texts under `<documents>`") → the summary-seed contract (A6 re-revised); `build_high_prompt`'s docstring updated likewise; the phase-106 D5 note (the `date` attribute) stays.
|
||||
5. `app/rag/retriever.py` — module docstring: the A7-revised paragraph ("the LLM receives the entire relevant document… the full text of the top-N documents is always fed through, never truncated") → the A6 re-revised contract: the retrieval path seeds SUMMARIES (the suggestion tier, locked A3); full text enters the context only through the capped `read` tool; `select_documents`/`select_documents_tiered` docstrings get the same one-line pointer (the functions themselves are untouched — they still serve the related tier + any remaining callers).
|
||||
6. `build_deflect_prompt` is UNTOUCHED (the LOW prompt + `DEFLECT_MODE` body stay byte-identical — the prompt lock pins it).
|
||||
7. ASSUMPTION: the section keeps the name `<documents>` (mock + copy contract) rather than a new `<suggested>` tag — the tag is a stability surface, the intro line carries the semantic change.
|
||||
|
||||
## Testing & Quality
|
||||
- `tests/unit/test_prompts.py` — a HIGH prompt built from docs with summaries: the block bodies are the summaries (sentinel: a full-content string that appears NOWHERE in the prompt; the summary text appears inside the `<document>` block), the intro line is present before the first block; the identity + `date` attributes byte-identical to today's format.
|
||||
- `tests/unit/test_prompts_dates.py` — the `date` attribute pins survive the body change (update bodies, keep the attribute assertions).
|
||||
- new: the preview fallback — a doc with `summary=None` → body = first 400 chars + `TRUNCATION_MARKER`; `summary=" "` → same; a short content (< 400) → the whole content, no marker; `BOR_SUGGESTION_PREVIEW_CHARS` env override honored.
|
||||
- new: the LOW prompt is byte-identical to pre-task on the same inputs (sha-pin via `build_deflect_prompt` output or the existing lock).
|
||||
- `tests/unit/test_prompt_lock.py` — if `SUGGEST_INTRO` is added as a constant, pin it there (sha + prefix); `PERSONA`/LOW-body anchors untouched.
|
||||
- `tests/unit/test_config.py` — the new setting.
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A grounded prompt built from 5 summary-bearing docs contains all 5 summaries + the intro, and ZERO full-content characters beyond the preview cap for NULL-summary docs.
|
||||
- [ ] The LOW prompt output is byte-identical to pre-phase for identical inputs.
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompts_dates.py tests/unit/test_prompt_lock.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 04 — `read` on a suggested doc adds its full text + the rewritten `read` copy
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content…" (suggested = "for reading" — the model reads what its summaries point at.)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`read` on a suggested document is allowed and adds the document's full text (all existing read mechanics unchanged); the dedupe refusal fires only for an already-READ document; the `read` tool description and the `TOOLS_SECTION` `read` clause are rewritten for the summary-seed mode; the prompt-lock anchors are re-cut.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `_execute_tool`'s `read` branch (~L1131–1136): `known = {(doc.source, doc.path) for doc in holder.read_docs}` — the `seed_docs` fall out of the dedupe set (a suggested doc is a summary in the prompt, NOT its full text). A first `read` of a suggested doc now succeeds through the EXISTING path unchanged: `holder.read_docs.append(doc)`, `holder.tool_calls += 1`, the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `holder.read_truncations` (→ the SSE `tool_result` frame), the `Document {source}/{path}:` header + phase-106 D5 `date:` line. A second `read` of the same (now full-text-in-context) document still gets `ALREADY_IN_CONTEXT` — its copy stays byte-identical (it already names the correct action: answer from the text in the prompt). The `seed_docs` parameter of `_execute_tool`/`run_agent` stays (callers unchanged) but its docstring + the `run_agent` docstring paragraph "``seed_docs`` are the documents the retrieval already put in context… re-reading one of them is rejected" are rewritten: the seeds are SUMMARY blocks in the prompt; reading one adds its full text; only an already-READ doc is refused.
|
||||
2. `app/rag/agent.py` — `AGENT_TOOLS` `read` function description: rewrite the leading contract sentences — the `<documents>` section shows the SUMMARIES of the top-ranked documents (their full texts are NOT in the prompt); use `read` to add one of them (or any other document) by its combined `source/path` string, exactly as shown in the `ls` output / the `<documents>` blocks; do not re-read a document you have already read — its full text is already in your prompt. KEEP byte-identical: the combined-`source/path` identity contract, the bare-path-will-not-resolve sentence, the truncation-notice paragraph, the "call one tool at a time" sentence, and the `path` parameter description (minus its "Only pass a document NOT already shown in the `<documents>` section" clause → "Do not re-read a document you have already read").
|
||||
3. `app/rag/prompts.py` — `TOOLS_SECTION`: rewrite ONLY the `read` clause the same way (today's "Do not call `read` for a document already shown in the `<documents>` section, even when the user asks you to open or read it — its full text is already in your prompt; answer directly from it." becomes: the `<documents>` section holds SUMMARIES — the full text of a suggested document is not in the prompt until you `read` it; do not re-read a document you have already read — its full text is already in your prompt; answer directly from it. If the user asks you to open or read a suggested document, `read` it — that is the point of the section.) KEEP byte-identical: the `ls` clause (the phase-94 drill-down contract), the `grep` clause, and the discipline rules (one call per reply, never repeat a refused/succeeded call, answer as soon as you have what you need).
|
||||
4. `tests/unit/test_prompt_lock.py` — re-cut the `TOOLS_SECTION` anchors for the new copy: `TOOLS_SECTION_SHA256` + `TOOLS_SECTION_LEN` ALWAYS change; `TOOLS_SECTION_PREFIX`/`SUFFIX` only if the new copy touches those exact boundaries (the prefix is the `ls`-clause opening and the suffix is the discipline-rules ending — both stay, so they should survive; verify, don't assume). Update the "Pre-phase-112 anchors" comment to cite the phase-118 re-cut (A6, owner directive 2026-09-15). `PERSONA` and the LOW-body anchors stay untouched — if they fail, the change leaked into locked text and the task is wrong.
|
||||
5. `app/rag/agent.py` — module docstring: the "A7 scope clarification" paragraph (~L120, "the top-2 seed documents stay whole — 'this should never happen'") → the A6 re-revised contract (seeds are summaries; the `read` path is the only full-text path and keeps its cap); point 3's seed-read refusal teaching → the already-read refusal.
|
||||
6. ASSUMPTION: the `ALREADY_IN_CONTEXT` constant text is unchanged (it is accurate for the already-read case and the E2E mock/telemetry key on stable copy).
|
||||
|
||||
## Testing & Quality
|
||||
- `tests/unit/test_agent.py` — update/extend:
|
||||
- a `read` of a `seed_docs` document SUCCEEDS: the full-content result (header + date line), `holder.read_docs` length 1, `holder.tool_calls` 1;
|
||||
- a second `read` of the same document → `ALREADY_IN_CONTEXT` (byte-identical line), counters untouched;
|
||||
- the cap path on a suggested doc (content > `read_max_chars`) → truncated result + notice + `read_truncations` entry (existing pins re-targeted from "seed" to "suggested");
|
||||
- the `AGENT_TOOLS` `read` description pins: the new summary contract present, the identity/truncation/one-call sentences byte-present;
|
||||
- `ls` and `grep` behaviors byte-identical (their existing pins untouched).
|
||||
- `tests/unit/test_prompts.py` — the `TOOLS_SECTION` inside a built HIGH prompt carries the new `read` clause; `ls`/`grep` clauses byte-identical (sha-pin the unchanged clauses if the file's style allows).
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `read('source/path')` on a suggested doc adds its full text to the context and counts as an executed call; a repeat is refused.
|
||||
- [ ] `PERSONA` + the LOW `DEFLECT_MODE` body byte-locks pass without re-cutting; only the `TOOLS_SECTION` anchors moved.
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Task 05 — `plan_turn` wiring: suggested seeding, citations, the durable record
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content… That way the LLM responds faster…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`plan_turn` seeds the HIGH branch with the top-5 suggested documents (summary blocks) instead of the top-2 full texts; the related tier becomes rank 6+; citations (`done.sources`) = suggested + agent-read; the durable record (`query_log` + log line) = suggested + related + read with a new `suggested=N` field; the LOW/deflection path stays byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/retriever.py` — new pure helper `select_related(chunks, excluded_ids: set[uuid.UUID], cap: int) -> list[Document]`: the SAME stable score-descending walk as `select_documents_tiered` (first-seen-chunk rank, dedupe by id), skipping documents in *excluded_ids*, no floor, at most *cap* documents. (The phase-113 related tier was "ranked after the cited tier"; with a contiguous top-5 suggestion set, "excluding the suggested" is exactly "rank 6+".) Unit-test it here or in task 05's tests — either location, one set of tests.
|
||||
2. `app/api/chat.py` — `plan_turn`: replace the `select_documents_tiered(chunks, n=settings.top_n_docs, floor=…, related_cap=…)` call with:
|
||||
```python
|
||||
suggested = select_suggested(chunks) # BOR_SUGGESTED_DOCS (5), no floor (A3)
|
||||
related_docs = select_related(chunks, {d.id for d in suggested}, settings.related_max_docs)
|
||||
```
|
||||
Both branches then use `suggested`/`related_docs`: the HIGH branch → `build_high_prompt(suggested, notes=…, kb_overview=…)`; the LOW branch → `build_deflect_prompt(titles, …)` UNCHANGED (weak-hit titles only). `summary_hits` = the hit chunks with `is_summary` whose parent document id is in the SUGGESTED set (redefined from the cited set — the counting code shape is unchanged).
|
||||
3. `app/api/chat.py` — `TurnPlan`: rename field `docs` → `suggested_docs` (SAME dataclass position; update every attribute access). Docstrings: the field holds the summary-seeded suggestion tier (A6); `related_docs` docstring updated (rank 6+ after the suggested set).
|
||||
4. `app/api/chat.py` — the `chat` endpoint's post-stream step: `run_agent(..., seed_docs=plan.suggested_docs, …)`; `cited_docs` loop over `[*plan.suggested_docs, *holder.read_docs]` (LOCKED A4 — the dedupe pattern is byte-identical); `record_docs` over `[*plan.suggested_docs, *plan.related_docs, *holder.read_docs]` (LOCKED A3 — the durable record keeps the full retrieval); `source_paths` unchanged.
|
||||
5. `app/api/chat.py` — the per-turn log line (PLAN §9): add `suggested=%d` (`len(plan.suggested_docs)`) after `summary_hits=%d`; the field order of every existing field is untouched (the phase-114 `retries=%d scaffold_stripped=%d` tail stays last).
|
||||
6. `app/config.py` — `top_n_docs` and `source_usefulness_floor` lose their seeding role (plan_turn was their only `app/` consumer — verified). KEEP both settings (env back-compat) and keep `select_documents`/`select_documents_tiered` (dormant public helpers): one-line docstring note on each — "phase 118 retired the full-text seeding role (A6); the suggested tier (`select_suggested`) seeds the prompt now." No deletions.
|
||||
7. ASSUMPTION: the `done.related` UI row (the phase-113 de-emphasized "nearby docs") keeps showing rank 6+ under the new tiering — the field name, shape and dedupe-vs-cited rule are unchanged (a read related doc stays a citation, never a "nearby doc").
|
||||
|
||||
## Testing & Quality
|
||||
- `tests/unit/test_chat_gate.py` — update/extend:
|
||||
- HIGH branch: the prompt contains the 5 suggested docs' SUMMARY text and NONE of their full content (sentinel); the related tier = rank 6+ (a 7-doc fixture: related = docs 6–7, capped by `related_max_docs`);
|
||||
- the deflected branch: prompt byte-identical to pre-task on the same chunks (sha-pin the output), suggestions/chips unchanged, `deflected=True`;
|
||||
- BOTH branches' TurnPlan carry suggested + related (the durable-record inputs);
|
||||
- `summary_hits` redefinition (a summary chunk whose parent is in the suggested set counts; one whose parent is related-only does not);
|
||||
- the log-line format pin (search the suite for the `summary_hits=` format string) gains `suggested=N` in the right slot;
|
||||
- settings: `suggested_docs`/`related_max_docs` honored; `top_n_docs`/`source_usefulness_floor` no longer consulted by `plan_turn` (behavioral pin, not a removal pin).
|
||||
- `tests/integration/test_chat_api.py` (and any sibling pinning the done-event sources) — `done.sources` = suggested + read (deduped) on a grounded turn with a mock `read` flow; `[]` on deflection; `done.related` = rank 6+ minus cited; `query_log.sources` = suggested + related + read.
|
||||
- `tests/unit/test_retriever.py` — `select_related` (walk order, exclusion, cap, no floor, determinism).
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A grounded turn's system prompt seeds exactly the top-5 suggested summaries; `run_agent` gets them as `seed_docs`.
|
||||
- [ ] `done.sources` = suggested + read (deduped, order-preserving); `query_log.sources` + the log line record suggested + related + read; `suggested=N` lands after `summary_hits=N`.
|
||||
- [ ] A deflected turn is byte-identical in prompt and frames to pre-phase.
|
||||
- [ ] `uv run pytest tests/unit/test_chat_gate.py tests/integration/test_chat_api.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 06 — E2E: the new summary-seed suite, the phase-24 E2E retired, old pins updated
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Playwright coverage of the summary-seed contract (summaries in the prompt, full text only via `read`, markdown docs summarized, citation chips = suggested + read); the retired phase-24 full-text E2E deleted; every E2E/integration pin of the retired behavior re-targeted to the new contract.
|
||||
|
||||
## Work
|
||||
1. NEW `tests/e2e/test_summary_seed_context.py` (run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`; DB up: `podman compose up -d db`). Pattern: the sibling suites' deterministic seeding (the mock LLM's bag-of-words `embed_text` + `import_sources` or the sync API — see `tests/e2e/test_read_truncation_cap.py`'s token-controlled seeding and `tests/e2e/test_whole_document_context.py`'s direct SQLAlchemy seeding; the mock's `SUMMARY_MODE` digest makes every stored summary deterministic, markdown included). Token-controlled bodies so the hybrid gate picks the intended suggested set deterministically. Cases:
|
||||
- **(a) summaries seeded**: the `END_OF_NOTES_TRIGGER` tail-echo question → the answer quotes the last 160 chars of the seeded `<documents>` context, which now end in the LAST suggested document's SUMMARY — pin the mock's summary digest text (the phase-30 digest shape), proving summaries reached the prompt.
|
||||
- **(b) full text NOT seeded**: the inverse of the phase-24 pin — a sentinel on a suggested document's LAST LINE is absent from the echoed context (the full content never reached the model pre-`read`).
|
||||
- **(c) `read` adds the full text**: a scripted mock tool flow `read`s a suggested document (the existing `_tool_flow` read-echo pattern, cf. `test_ls_tree_drilldown.py`) → the answer quotes the full-text tail from the read RESULT (the sentinel now appears, sourced from the read, not the seed).
|
||||
- **(d) markdown gets a summary**: a markdown document in the KB appears as a suggestion block carrying the mock `SUMMARY_MODE` digest (not a content preview) — the locked A2 end-to-end proof.
|
||||
- **(e) citation chips**: after case (c), the UI chip row = suggested + the read document (deduped — the read doc is among the suggested, so chips = the suggested set), each a `done.sources` entry; `done.related` renders the de-emphasized row (rank 6+) when the seeded KB is large enough to have one.
|
||||
2. `tests/e2e/test_whole_document_context.py` — DELETE (the retired full-text-seed contract; locked A1/A6 — no kill switch to re-scope it to).
|
||||
3. `tests/e2e/test_ls_tree_drilldown.py` — the `READ_ANSWER_PREFIX` flow (~L61–67, L229–234, L660–666): today the scripted read targets a top-2 seed and is REFUSED (`ALREADY_IN_CONTEXT`), the mock answering from the `<documents>` full text. Re-target: the scripted read target stays, but the read now SUCCEEDS (seeds are summaries) and the mock answers from the READ RESULT with the same citation shape — update the prefixes/quotes to key off the read result (the full text, first 80 chars — the content is now delivered by the read, so the quote is byte-identical to today's) and the comments to the new contract. If the mock's answer-from-`<documents>` branch (its `ALREADY_IN_CONTEXT`-keyed flow, `mock_llm.py` ~L237) is no longer exercised by ANY suite, keep it (the already-read refusal is still real) but update its docstring (it now describes the already-read case, not the top-2-seed case).
|
||||
4. `tests/e2e/test_read_truncation_cap.py` — the setup guard (~L364–379) that asserts the scripted read target is NOT a top-2 seed (its only reason was the seed-read refusal) — REMOVE the guard and its docstring; a seed-target read now executes, so the cap fires either way.
|
||||
5. `tests/e2e/test_document_summaries.py` — the phase-30 pins: `summary.summaries == 1` for the yaml + md-control pair (~L169) → `== 2` (markdown summarized too, locked A2); the "md control has no summary" assertions → inverted to "the md control HAS the mock digest summary"; the summary-chunk count pins gain the md doc's chunk.
|
||||
6. `tests/e2e/mock_llm.py` — extend ONLY if a case above needs a new trigger; the marker contracts stay (`<tools>` presence, `DEFLECT_MODE`, `_READ_RESULT_PREFIX` header, `END_OF_NOTES_TRIGGER`, `SUMMARY_MODE`). Update the docstrings that describe the top-2-seed/`ALREADY_IN_CONTEXT` scenario (~L235–241).
|
||||
7. Audit sweep (markers, in `tests/e2e/`, `tests/integration/`, `tests/unit/`): `ALREADY_IN_CONTEXT`, "already shown in the `<documents>`", "top-2 seed", full-content-in-prompt sentinels, `summary.summaries` counts, the per-turn log-line format (`summary_hits=` — gains `suggested=N`). Every hit: pin of RETIRED behavior → re-target to the new contract; pin of UNCHANGED behavior → leave. `tests/unit/test_agent.py`'s seed-read pins were already updated in task 04 — do not duplicate.
|
||||
|
||||
## Testing & Quality
|
||||
- The new suite green in isolation (the command above); every re-targeted suite green; full suite green.
|
||||
- Coverage: **>90%** on `app/` (E2E is uncounted; keep the unit coverage from tasks 01–05 intact).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov` green (cases a–e).
|
||||
- [ ] `tests/e2e/test_whole_document_context.py` no longer exists; `tests/e2e/test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`, `test_document_summaries.py` green with the new-contract pins.
|
||||
- [ ] The audit sweep left no test pinning the retired full-text-seed or seed-read-refusal behavior.
|
||||
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,23 @@
|
||||
# Task 07 — The fixture-battery gate re-run (real model) + verdict recorded
|
||||
|
||||
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…That way the LLM responds faster…" (the summary-seed behavior must be proven against the real configured chat model before the phase completes — the phase-70/72/94 tool-copy gate precedent.)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The fixture KB dump is rebuilt so every fixture document (all eight are markdown) carries a stored summary — the phase-118 end state — and the real-model fixture battery passes against it under the locked gate semantics (locked A7), with the verdicts recorded in `TOOL_CALLING_TESTING.md`.
|
||||
|
||||
## Work
|
||||
1. Rebuild the fixture dump — `uv run python -m scripts.load_test_kb` (the real pipeline: `import_sources` now summarizes every document, task 01 — the eight markdown fixture docs gain `doc.summary` + one embedded `is_summary` chunk each, so the dump's documents/chunks content changes; a rebuild is required, this is a legitimate re-run trigger — "the import pipeline's summary policy changed (phase 118)"). Preconditions: DB up, the aipi `embed` + `lite` endpoints reachable (`.env`). Verify from the build output: `summary_errors=0`, zero failed folder-summary batches, and the retrieval report's battery all-GROUNDED (the honesty gate is untouched by this phase, so grounded/deflected should be unchanged; if a question now deflects, it is a fixture-content regression — fix the fixture anchor per the build's own warning, never the gate). Commit the regenerated `tests/fixtures/test_kb.dump.sql` with the phase (it is tracked).
|
||||
2. Run the gate — `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` (the fast loop: inline restore + the curated battery against the live endpoint, the configured chat model). Use `--turns N` only for the copy-iteration micro-loop while refining task 04's `read` copy; the VERDICT run is always the full battery.
|
||||
3. Interpret the verdict under locked A7 (owner-confirmed 2026-09-15): the phase gate = condition 1 (all turns answer) AND condition 2 (zero round-cap hits) AND condition 4 (fixture-mode contract accuracy ≥ 0.90) — the script prints each; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is the intended fast path now, not tool abandonment. If the script exits 1 on condition 3 alone with 1/2/4 green, the phase gate is PASS — record exactly that (the four metrics + the wall time + the `contract X/Y` line, the TOOL_CALLING_TESTING.md verdict format).
|
||||
4. Copy iteration (only if condition 4 degrades or condition 2 is hit): refine the `read` description (`app/rag/agent.py` `AGENT_TOOLS`) / `TOOLS_SECTION` clause (`app/rag/prompts.py`) — re-cut the `test_prompt_lock.py` `TOOLS_SECTION` anchors again if the copy moves — micro-loop with `--turns`, then a full verdict run. `classify_call` needs NO re-baselining: a `read` of a suggested document is a well-formed call targeting a resolvable document — contract-correct under both old and new app semantics (its docstring's `ALREADY_IN_CONTEXT` note already excludes dedupe re-reads from the metric).
|
||||
5. `TOOL_CALLING_TESTING.md` — append a dated section ("Phase 118 — summary-seed gate, 2026-09-15…"): the four conditions + metrics, the contract line, the wall time, the model name, the dump's rebuild note (summaries now on all eight docs), the locked-A7 semantics note (condition 3 reported, not gated), and the `read` copy's final shape (or a pointer to it). The controlled-methodology sections above stay untouched (append-only history).
|
||||
|
||||
## Testing & Quality
|
||||
- The gate is the test: no unit suite here; the rebuild's own round-trip verification (dump restore + per-table checksum) must pass, and the full `uv run pytest` must stay green after the dump regeneration (E2E suites that restore the fixture dump pick up the summary rows — if any suite pins the old 9-chunk fixture state, that pin moves to the new 17-chunk state: 9 content + 8 summary chunks).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `tests/fixtures/test_kb.dump.sql` regenerated with all eight fixture docs summarized (round-trip checksum verified by the build).
|
||||
- [ ] Full-battery verdict run recorded: conditions 1, 2, 4 GREEN (condition 3 reported); the telemetry + semantics note appended to `TOOL_CALLING_TESTING.md`.
|
||||
- [ ] `uv run pytest` green with the new dump (including any fixture-chunk-count pins updated).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -1,3 +1 @@
|
||||
# TODO
|
||||
|
||||
- [ ] I want to change the way the initial chat context works. Right now we feed two entire docments into context before the LLM responds. That was a good idea before we had complex tools and document summaries, but now we have faster ways for the LLM to identify what information it needs without reading 2 files. Moving forward, after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content - basically a "start here if these summaries seem right to you" approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant because the similarity search had a bad day.
|
||||
Reference in New Issue
Block a user