chore(agent): phase roadmap from TODO.md, 5 phases
Protocol B append (phases 141-145) for the five TODO items: the read-range dedupe false "Already in context" bug (141), the stale KB overview recency ordering + honest intro (142), the mobile keyboard pop after answer completion (143), the document modal back-button history entry (144), and the git original-source links in answers + the doc modal (145). TODO.md cleared (items now live in .agents/phases/todo/). LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,36 @@
|
||||
# Phase 142 — the KB overview stays fresh: newest documents survive the cap, and the model is told the outline is partial
|
||||
|
||||
**Source:** `TODO.md` L4 — "The agent is really struggling with questions like 'Summarize the last 10 phases of brain of Reese'. When I asked that to the current deployment it said 'I'll pull the details for the last 10 phases (104–113).' when it clearly had phase 137 loaded in the RAG… 'The <knowledge_base> section in the system prompt only generated summaries up to phase 113 at import time. The phases 128–137 exist as full markdown files… but their summaries weren't included in the initial KB index. This is a KB sync/generation gap, not a retrieval logic bug.'"
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/overview.py` — `regenerate_overview` (the document-list query `order_by(Document.source, Document.path)` ~L232, best-effort change-gated per the module docstring), `build_overview_prompt` (one line per doc `source — path — title — {first summary line}`, capped at `max_chars` with the shared `TRUNCATION_MARKER`), `load_kb_overview`; `app/config.py` L216 — `overview_input_max_chars` (`BOR_OVERVIEW_INPUT_MAX_CHARS`, default 40 000); `app/rag/prompts.py` — `_KB_INTRO` (L128–132: literally says "(generated at import time)") and `build_kb_section` (L399+); `app/api/sync.py` ~L590 — the overview regeneration gate (`summary.added + summary.updated > 0` — the sync IS change-gated correctly; the staleness is inside the generation, not the trigger); `tests/e2e/test_kb_overview.py` (the mock's `KB_OVERVIEW_MODE` digest = first 8 tokens of the document list — ORDER-DEPENDENT, must be re-pinned), `tests/integration/test_import_docs_overview.py` (the import-time trigger), `app/models.py` — `Document.created_at` (L123 — the DOCUMENT's date, phase 106: git-history/mtime-derived and owner-correctable, the same date the `read` result's `date:` line shows — the "when is this document from" signal, exactly what "the last N …" means). Root cause (traced): on the owner's 1 000-doc KB the document list is ≈100k+ chars; the 40k cap cuts a LEXICOGRAPHIC prefix (`order_by (source, path)` — "113_…" sorts before "128_…", and "131_…"–"140_" sort after "13_…"), so the newest phases never reach the lite model and the stored outline confidently describes a KB that ends mid-11x; the model then anchors "the last 10 phases" on that stale image.
|
||||
**Locked (owner-confirmed 2026-09-30, chat):** recency ordering (newest documents first so they survive the cap; the cap itself is unchanged) + the honest `<knowledge_base>` intro (drops "(generated at import time)"; tells the model the outline is compact/possibly partial and to verify "latest/complete" questions with `ls`/`grep` first). No map-reduce rewrite this phase.
|
||||
|
||||
## Objective
|
||||
"Summarize the last 10 phases" stops being a guess: the overview's input is ordered newest-first so recent documents are the ones the outline covers when the cap bites, and the `<knowledge_base>` intro stops teaching the model that a partial outline is the whole KB — it now says the outline is a possibly-partial picture and points at `ls`/`grep` for "what's the latest / the complete set" questions.
|
||||
|
||||
## Dependencies
|
||||
- `141_read_range_dedupe_fix` (todo) — pipeline order only; no code dependency.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Ordering (task 02):** `regenerate_overview`'s query becomes `order_by(Document.created_at.desc(), Document.source, Document.path)` — newest documents FIRST (`created_at` is the document's date per phase 106 — the same date the `date:` line in a read result shows; the owner's phase-128–137 files carry newer git dates than the 113-era files, so they lead the list and survive the cap). The (source, path) secondary keys keep the tie-break deterministic for same-date docs (the common case within one commit/import — those fall back to the old lexicographic order, so same-date fixtures stay byte-identical). The cap (`BOR_OVERVIEW_INPUT_MAX_CHARS`, default 40 000) and the cap's cut+`TRUNCATION_MARKER` behavior are UNCHANGED (house truncation convention; the env knob stays the owner's lever). Zero other call-site changes: the sync's change-gate (task-31/32 trigger) already regenerates whenever the KB changes — the staleness was purely inside the list.
|
||||
- **Intro (task 03):** `_KB_INTRO` rewritten (it is a prompt constant — PLAN §4's convention: persona/prompt text changes ship with the phase's prompt tests; the `<knowledge_base>` TAG and the section's cap behavior in `build_kb_section` stay byte-stable): the outline is a COMPACT, possibly partial picture of the KB that favors RECENT documents — and for questions about the latest state or the complete set ("the last N …", "everything about …"), verify with `ls` (drill down) / `grep` before answering from the outline. One short paragraph, same tone as `SUGGEST_INTRO` (L152).
|
||||
- **Repro (task 01):** before any fix, a failing test proves the traced root cause: `build_overview_prompt` given a doc list whose joined lines exceed `max_chars` keeps the lexicographic FIRST lines and drops the last; and (integration) a KB where the newest docs' paths sort lexicographically LAST is regenerated with the pre-fix ordering → the stored outline (the mock's 8-token digest) names the OLD docs, not the new ones. The repro is written against the CURRENT code (red), then tasks 02/03 flip it green.
|
||||
- **E2E (task 04):** dedicated file — a fixture KB whose document count × line length exceeds a FORCED small cap (`BOR_OVERVIEW_INPUT_MAX_CHARS` set low via the E2E settings fixture) → regenerate through the mock → the stored outline's digest tokens come from the NEWEST document (recency ordering visible end-to-end through the real prompt path), and the served chat prompt's `<knowledge_base>` section carries the NEW intro copy (asserted from the echo path or the direct prompt-build surface — the phase-31 E2E already exposes the section to the UI via the first-bullet echo).
|
||||
|
||||
## Tasks
|
||||
1. `01_cap_ordering_repro.md` — the red repro tests (unit: cap keeps the lexicographic prefix; integration: newest-sorting-last docs missing from the regenerated outline).
|
||||
2. `02_recency_ordering.md` — the `created_at.desc()` ordering + the order-dependent digest re-pins.
|
||||
3. `03_honest_kb_intro.md` — the `_KB_INTRO` rewrite + the prompt tests.
|
||||
4. `04_e2e_outline_freshness.md` — the dedicated E2E (forced cap → newest doc in the outline; the new intro served).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `build_overview_prompt` cap behavior unchanged (the cut + marker, the pathological budget); the new ordering (regeneration with mixed `created_at` docs → newest first; same-date tie-break by (source, path)); the intro copy pin.
|
||||
- Integration: the import-time trigger still regenerates exactly on KB change (the pre-existing `tests/integration/test_import_docs_overview.py` green); the repro flipped green.
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only; `tests/e2e/test_kb_overview.py` (phase 31) green with re-pinned digest.
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] with the cap forced, the regenerated outline is built from the NEWEST documents (the 1 000-doc "ends at phase 113" failure mode is gone: the newest sort-last docs are the ones represented)
|
||||
- [ ] the served `<knowledge_base>` section carries the new honest intro (no "(generated at import time)"; the verify-with-`ls`/`grep` teaching present)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_kb_overview_freshness.py -v --no-cov` green in isolation; `tests/e2e/test_kb_overview.py` still green in isolation; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change in completed phases (the cap env knob, the change-gated trigger, the `build_kb_section` cap/marker behavior, the deflect path)
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 01 — the red repro: the cap keeps a lexicographic prefix, so newest docs are the ones lost
|
||||
|
||||
**Phase:** `142_kb_overview_freshness` · **Source:** `TODO.md:4` — "The agent is really struggling with questions like 'Summarize the last 10 phases of brain of Reese'. When I asked that to the current deployment it said 'I'll pull the details for the last 10 phases (104–113).' when it clearly had phase 137 loaded in the RAG… this is a KB sync/generation gap, not a retrieval logic bug."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Lock the traced root cause as failing tests BEFORE the fix: `build_overview_prompt` cuts the document list at the cap keeping the FIRST lines (which, under the current `order_by (source, path)`, are the lexicographically first — NOT the newest), and a regeneration on a KB whose newest documents sort lexicographically LAST produces an outline that names the old documents. These tests are red on the current code and turn green in tasks 02/03.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_overview.py` (the phase-31 unit home — if the overview units live elsewhere, e.g. inside `tests/unit/test_agent.py`'s neighbors, use the existing file for `build_overview_prompt`'s pins) — the cap-ordering repro, unit level, no LLM:
|
||||
- build 20 fake doc rows whose `source — path` prefixes are `s — 01_old_a.md` … `s — 20_new_z.md` (a path ordering that ALSO puts the "newest" last — the owner's phase-dir shape: names like `113_…` vs `128_…` vs `131_…`), each line ~100 chars, `max_chars=500` → the prompt's user message holds ONLY the first lines (assert: `01_old_a` present, `20_new_z` ABSENT, the `TRUNCATION_MARKER` line present — the current behavior pinned as-is);
|
||||
- the assertion this task ADDS is the one that will flip: a helper-level check that documents the INTENT — the list reaching the model should lead with the NEWEST rows — written as the expected post-fix order (fails pre-fix).
|
||||
2. `tests/integration/test_import_docs_overview.py` (or its sibling for regeneration — the file that already drives `regenerate_overview` against the mock) — the KB-shape repro, integration level:
|
||||
- a fixture KB shaped like the owner's: one source, folders/paths whose names sort lexicographically BEHIND their content's recency (e.g. an `old/…` batch + a `zz_new/…` batch whose paths sort LAST). The batches' `created_at` (the document date — phase 106 derives it from the file's mtime for a local fixture) must differ: set the files' mtimes (`os.utime`) before import so the old batch is genuinely older, OR update the old batch's `created_at` to a past date in a raw SQL step after import — the test owns its fixture, either seam is fine,
|
||||
- regenerate through the mock `KB_OVERVIEW_MODE` (the 8-token digest of the document list, `tests/e2e/mock_llm.py` ~L2906) with a small `BOR_OVERVIEW_INPUT_MAX_CHARS` so the cap bites,
|
||||
- assert the CURRENT failure: the stored outline's tokens come from the lexicographically-first (oldest) documents, NOT the newest — the exact "summaries only up to phase 113" shape, reproducible in a test.
|
||||
- Mark both tests with the outcome they expect POST-fix (newest docs win); they FAIL now — that red is the repro. (Task 02 makes them green.)
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: the two repro tests above — RED on the current code (asserted by running them in this task), green after task 02 (+ task 03 for the intro).
|
||||
- Coverage: **>90%** on this task's new/modified code (tests — coverage applies to `app/` as a whole).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the two repro tests exist and FAIL on the pre-fix code for the documented reason (lexicographic-prefix cap; oldest docs in the outline) — the failure output names the root cause
|
||||
- [ ] no application code changed in this task (the repro is tests-only)
|
||||
- [ ] the rest of `uv run pytest` stays green (the repro tests are the only new failures)
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — recency ordering: newest documents first in the overview's input
|
||||
|
||||
**Phase:** `142_kb_overview_freshness` · **Source:** `TODO.md:4` — "The agent is really struggling with questions like 'Summarize the last 10 phases of brain of Reese'. When I asked that to the current deployment it said 'I'll pull the details for the last 10 phases (104–113).' when it clearly had phase 137 loaded in the RAG… this is a KB sync/generation gap, not a retrieval logic bug."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`regenerate_overview` builds the lite model's document list newest-first, so when `BOR_OVERVIEW_INPUT_MAX_CHARS` cuts the list, the documents that fall off are the OLDEST — and the outline (the model's a-priori KB picture) always represents the recent state of the KB. Task 01's red repro turns green.
|
||||
|
||||
## Work
|
||||
1. `app/rag/overview.py` — `regenerate_overview`'s document-list query: `order_by(Document.created_at.desc(), Document.source, Document.path)`. Update the query's inline comment + the module docstring (the "one line per document" description): the ordering is RECENT-FIRST by the DOCUMENT's date (phase 106's `created_at` — git-history/mtime-derived, owner-correctable, `created_at_manual` rows included exactly as stored: the ordering reads the column, it never re-derives dates; the (source, path) tie-break keeps same-date docs in the old lexicographic order — deterministic for the mock's digest, and byte-identical for single-date fixtures). The cap, the cap's cut + `TRUNCATION_MARKER`, the zero-docs no-op, the `LLMError` fail-soft, the upsert — ALL unchanged.
|
||||
2. `tests/unit/test_overview.py` — the unit pins:
|
||||
- the task-01 cap test flips: with mixed `created_at` rows (the row shape `(source, path, title, summary)` is order-agnostic — the ORDER comes from the query, so the unit test for the QUERY ordering goes against the DB: a small session with 6 docs in two `created_at` batches → the SELECT returns newest batch first, (source, path) within a batch),
|
||||
- the byte-identical pins for `build_overview_prompt` itself are UNTOUCHED (it still receives rows in whatever order the caller passes — its contract is order-preserving; only the CALLER's order changes).
|
||||
3. `tests/integration/test_import_docs_overview.py` — the integration repro flips green: the second (newer) import batch's paths now lead the digest.
|
||||
4. Re-pin the ORDER-DEPENDENT digest expectations that assumed the old order (the mock's 8 tokens are the first document line's tokens — under recency-first that is now the NEWEST doc's line):
|
||||
- `tests/e2e/test_kb_overview.py` — `test_mock_generated_outline_is_stored_and_echoed`'s expected digest (compute it from the fixture's newest document, not the lexicographic first — the test's fixture documents and their `created_at` values decide; the test seeds its row directly for tests 1–3, so only test 4's regeneration digest is order-sensitive),
|
||||
- any other pin asserting a specific first line of a regenerated overview (grep the suite for the digest literal — re-pin where the KB has more than one document with differing `created_at`; single-batch fixtures where every doc shares a `created_at` stay byte-identical by the (source, path) tie-break).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: the ordering tests + the flipped repro (task 01) + the re-pinned digests.
|
||||
- Regression: the phase-31 `KB_OVERVIEW_MODE` branch in `tests/e2e/mock_llm.py` is UNCHANGED (it digests whatever list the app sends — the mock is order-agnostic by construction); `tests/e2e/test_kb_overview.py` green in isolation after the re-pins.
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] task 01's two repro tests are GREEN (the newest documents lead the capped list; the outline represents the recent KB)
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green in isolation; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change in completed phases beyond the list ORDER (the cap env knob, the trigger gate, the fail-soft, the upsert, the mock's generation branch — all byte-identical)
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 03 — the honest `<knowledge_base>` intro: the outline is partial, and "latest" questions verify first
|
||||
|
||||
**Phase:** `142_kb_overview_freshness` · **Source:** `TODO.md:4` — "The agent is really struggling with questions like 'Summarize the last 10 phases of brain of Reese'. When I asked that to the current deployment it said 'I'll pull the details for the last 10 phases (104–113).' when it clearly had phase 137 loaded in the RAG…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The model stops treating the outline as the KB's full, import-time-frozen inventory: `_KB_INTRO` says what the outline actually is (a compact, possibly partial picture that favors recent documents) and gives the one behavior that fixes the owner's question class — for "latest / complete set" questions, verify with the tools (`ls` drill-down / `grep`) before answering from the outline.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — rewrite `_KB_INTRO` (L128–132). Current: `"The basic categories of everything in this knowledge base (generated at import time):\n"`. New copy (short — the section is char-budgeted by `build_kb_section`; keep it 1–2 lines): the outline is a COMPACT picture of the knowledge base's categories and topics that FAVORS RECENT DOCUMENTS and may omit older or lesser ones — it is a map of what exists, not a complete inventory; for questions about the LATEST state or the COMPLETE set ("the last N …", "every …", "all the …"), verify with `ls` (drill into the folders) or `grep` before answering from the outline. The "(generated at import time)" phrase is GONE (it is false — the sync regenerates the row on KB change — and it taught the model to discount/freeze the section).
|
||||
2. `app/rag/prompts.py` — leave `build_kb_section` (L399+) byte-stable: same tag, same cap loop, same pathological-budget behavior — ONLY the intro constant changes. Update the constant's doc comment (the phase-31 note) with this phase's rationale in one line.
|
||||
3. Prompt tests — the house prompt-suite pins (`tests/unit/` — the file pinning `build_kb_section` / the HIGH prompt's section order, the phase-31 pins):
|
||||
- the new intro copy pinned verbatim (the exact string),
|
||||
- a no-overview prompt stays byte-identical to the pre-phase text (the phase-15 convention — the existing pin passes unchanged: an empty overview omits the whole section),
|
||||
- the section's cap/marker behavior pins pass unchanged (the intro is inside the budget loop — a one-line-longer intro changes the pathological-budget edge only if a test pinned the exact cut against the OLD intro length; re-pin any such test to the new intro length — the BEHAVIOR is unchanged),
|
||||
- the HIGH prompt's section-order pin (`<relevance>` → `<knowledge_base>` → `<tuning>` → body) passes unchanged.
|
||||
4. The deflect path is UNTOUCHED (the LOW prompt carries no `<knowledge_base>` section — the existing byte-identity pins pass as-is).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the intro copy pin; the section-build pins (cap, marker, pathological budget, empty-omission); the section-order pin.
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `_KB_INTRO` no longer contains "(generated at import time)"; it names the recency bias and the verify-with-`ls`/`grep` behavior (pinned verbatim)
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change in completed phases (the section's tag/cap/marker, the deflect prompt's byte-identity, the section order)
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 04 — the dedicated E2E: the regenerated outline leads with the newest document, and the new intro is served
|
||||
|
||||
**Phase:** `142_kb_overview_freshness` · **Source:** `TODO.md:4` — "The agent is really struggling with questions like 'Summarize the last 10 phases of brain of Reese'. When I asked that to the current deployment it said 'I'll pull the details for the last 10 phases (104–113).' when it clearly had phase 137 loaded in the RAG…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The browser-visible proof of the whole phase: with the cap forced to bite, a sync-path regeneration produces a stored outline whose digest comes from the NEWEST document (recency ordering, end-to-end through the real prompt path), and the served chat prompt's `<knowledge_base>` section carries the new honest intro.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_kb_overview_freshness.py` — dedicated Playwright file, run in isolation, mock-only (the house conventions of `tests/e2e/test_kb_overview.py` — per-module truncate/re-import, the settings override fixture, the login helper, the SSE echo assertions):
|
||||
- fixture: one source, TWO batches of documents whose document DATES differ (the old batch's files get past mtimes via `os.utime` before import, the new batch is current — phase 106's mtime fallback for local sources; the new batch's PATHS sort lexicographically LAST, e.g. `old/a.md`… vs `zz_new/z.md` — so the pre-fix lexicographic order and the post-fix recency order disagree about which document is first),
|
||||
- settings: `BOR_OVERVIEW_INPUT_MAX_CHARS` set small enough (via the E2E settings override) that the document list is CUT — the cap must bite or the ordering is unobservable;
|
||||
- test 1 (`test_regenerated_outline_leads_with_newest_doc`): import, then drive the regeneration through the mock the same way `test_kb_overview.py` test 4 does (the real `regenerate_overview` against the mock's `KB_OVERVIEW_MODE` digest) → the stored `kb_overview` row's 8 digest tokens come from the NEWEST batch's first (recency-ordered) line — assert the exact tokens (computed from the fixture in the test, not hardcoded wording);
|
||||
- test 2 (`test_kb_section_carries_honest_intro`): the intro is a prompt constant — the deterministic surface is the section builder itself, called IN the E2E file the same way this file already calls `regenerate_overview`/`build_overview_prompt` directly against the app (the phase-31 file's own convention): `build_kb_section(<a seeded multi-line outline>)` → assert the section's lead line IS the new intro verbatim (no "generated at import time"), the tag + cap behavior intact; the UI injection path (the section reaching a rendered turn) is the phase-31 first-bullet echo, which a third lightweight test re-asserts here against the SEEDDED row so this file stands alone on the injection surface too;
|
||||
- the pre-existing phase-31 file is NOT modified here (its re-pins landed in task 02).
|
||||
2. Run the regression set in isolation (each file its own run — the suite-isolation contract): `test_kb_overview_freshness.py` (new), `test_kb_overview.py` (phase 31, task-02 re-pins).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: `uv run pytest tests/e2e/test_kb_overview_freshness.py -v --no-cov` — green IN ISOLATION (DB up: `podman compose up -d db`), mock-only.
|
||||
- Integration: the intro-observation fallback (if test 2 takes the `build_kb_section` surface) lives in the unit/integration suite, not the E2E.
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] with the cap biting, the regenerated outline's digest tokens come from the NEWEST document (the owner's "ends at phase 113" shape is impossible — the newest docs lead)
|
||||
- [ ] the new intro copy is pinned verbatim as the section's lead line (the `build_kb_section` surface), and the section still reaches a rendered turn (the phase-31 echo path)
|
||||
- [ ] `uv run pytest tests/e2e/test_kb_overview_freshness.py -v --no-cov` green in isolation; `tests/e2e/test_kb_overview.py` still green in isolation
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change in completed phases (the phase-31 injection path, the mock's generation branch)
|
||||
Reference in New Issue
Block a user