phase: 142_kb_overview_freshness

All verification complete. Final report:

**Phase 142 final verification pass — all green**

- **Verified implemented** (working tree matches design; app diff is exactly 2 lines of logic): `app/rag/overview.py` — `regenerate_overview` now orders the lite-model document list `created_at` desc, `(source, path)` tie-break (newest docs survive the cap); `app/rag/prompts.py` — `_KB_INTRO` rewritten to the honest intro (recency bias, "map not inventory", verify-with-`ls`/`grep`; "(generated at import time)" gone).
- **Completion criteria:**
  1. Forced-cap outline leads with newest docs — ✅ E2E `test_kb_overview_freshness.py` test 1 (cap 300, digest from `zz_new/z.md`, no "old" tokens) + integration repro (cap 150 → `mydocs zz new 03 md zz new 03`)
  2. New honest intro served in `<knowledge_base>` — ✅ unit pin (verbatim) + E2E test 2 (`build_kb_section` surface, tag/cap byte-stable)
  3. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3359 passed**, `app/` TOTAL **99%** (>90%); `uv run pytest tests/e2e/test_kb_overview_freshness.py -v --no-cov`: **3 passed** in isolation; `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov`: **4 passed** in isolation; `uv run ruff check . && uv run pyright`: **clean, 0 errors**
  4. No behavior change in completed phases — ✅ cap env knob, change-gated sync trigger, `build_kb_section` cap/marker, deflect path all byte-identical (pinned pins pass in the 3359)
- **Documentation:** README.md reviewed in full (every `.env.example` var documented with matching defaults, all endpoints/routes verified against `app/api/`, Quick Start/deploy/troubleshooting claims verified) and AGENTS.md re-reviewed (rules + quick reference accurate) — **verified consistent, no changes needed**.
- **Decisions:** none required — no defects found; no live-infra changes (only the already-running local `db` container was used by tests).
- **Next pending phase:** `.agents/phases/todo/143_mobile_keyboard_after_answer`.

tests/integration/test_import_docs_overview.py

@@ -38,19 +38,22 @@ calls on a fresh import).
 from __future__ import annotations
 
 import logging
+import os
 from collections.abc import Iterator
+from datetime import UTC, datetime, timedelta
 from pathlib import Path
 
 import pytest
 from sqlalchemy import text
 from sqlalchemy.orm import Session
 
-from app.config import Settings
+from app.config import Settings, get_settings
 from app.db import SessionLocal, db_available
 from app.models import KbOverview
 from app.rag.llm import LLMError
 from app.rag.sources_meta import current_sources_version
 from scripts import import_docs
+from tests.e2e.mock_llm import TOKEN_RE
 from tests.fakes import FakeEmbedder
 
 
@@ -62,6 +65,26 @@ class FailingChatEmbedder(FakeEmbedder):
         raise LLMError("simulated lite-model failure (test sentinel)")
 
 
+class OverviewDigestEmbedder(FakeEmbedder):
+    """A ``lite`` stand-in reproducing the E2E mock's ``KB_OVERVIEW_MODE``
+    branch (``tests/e2e/mock_llm.py``): the overview call returns the
+    document list's first 8 tokens (lowercased, the mock's
+    ``TOKEN_RE``) as the outline. The digest is ORDER-DEPENDENT, so the
+    stored row names exactly the documents that reached the model — the
+    phase 142 repro's witness. Every other call falls through to
+    :class:`FakeEmbedder`'s deterministic ``Summary of …`` digest."""
+
+    async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str:
+        system = next((m["content"] for m in messages if m.get("role") == "system"), "")
+        if "KB_OVERVIEW_MODE" in system:
+            self.chat_calls.append(list(messages))  # the base appends only on its path
+            user = next((m["content"] for m in messages if m.get("role") == "user"), "")
+            return "Knowledge base outline:\n- " + " ".join(
+                TOKEN_RE.findall(user.lower())[:8]
+            )
+        return await super().chat(messages, model)
+
+
 def _row(db: Session) -> KbOverview | None:
     """The stored ``kb_overview`` row (freshly reloaded)."""
     db.expire_all()
@@ -108,6 +131,41 @@ def src(tmp_path: Path) -> Path:
     return root
 
 
+@pytest.fixture()
+def owner_kb(tmp_path: Path) -> Path:
+    """Phase 142, task 01 — a KB shaped like the owner's: one source, an
+    OLDER batch whose paths sort FIRST (``old/01..03.md``) and a NEWER
+    batch whose paths sort LAST (``zz_new/01..03.md``) — the phase-dir
+    shape where ``131_…`` sorts after ``113_…``. Both batches get
+    explicit, staggered file mtimes (``os.utime``): phase 106 sources
+    ``created_at`` from the local file's mtime, so recency strictly
+    OPPOSES the lexicographic path order (old/01 is the oldest and sorts
+    first; zz_new/03 is the newest and sorts last). The dates are
+    RELATIVE to the run (old batch ≈ 90 days back, new batch ≈ 1–3 days
+    back, each file strictly later than the previous): they stay in the
+    past forever (a fixed calendar date would drift beyond
+    ``normalize_doc_date``'s future-skew tolerance and collapse to
+    "today") while the staggered order — and thus the digest — is a
+    pure function of the fixture, stable across runs."""
+    root = tmp_path / "MyDocs"
+    old = root / "old"
+    new = root / "zz_new"
+    old.mkdir(parents=True)
+    new.mkdir()
+    now = datetime.now(UTC)
+    for i in (1, 2, 3):
+        f = old / f"{i:02d}.md"
+        f.write_text(f"# Old {i:02d}\n\nOld body {i}.\n", encoding="utf-8")
+        ts = (now - timedelta(days=92 - i)).timestamp()
+        os.utime(f, (ts, ts))
+    for i in (1, 2, 3):
+        f = new / f"{i:02d}.md"
+        f.write_text(f"# Zz New {i:02d}\n\nNew body {i}.\n", encoding="utf-8")
+        ts = (now - timedelta(days=4 - i)).timestamp()
+        os.utime(f, (ts, ts))
+    return root
+
+
 @pytest.fixture(autouse=True)
 def _clean_kb(db: Session) -> Iterator[None]:
     db.execute(text("TRUNCATE chunks, documents, kb_overview, folder_summaries"))
@@ -349,3 +407,60 @@ def test_prune_only_run_bumps_sources_version(
     )
     assert _version(db) == 2  # the prune-only change bumped exactly once
     assert _row(db) is not None  # the outline row is untouched
+
+
+# ---------- phase 142, task 01: the KB-shape repro (red pre-fix) ----------
+
+
+def test_overview_outline_names_the_newest_docs_when_the_cap_bites(
+    db: Session,
+    owner_kb: Path,
+    monkeypatch: pytest.MonkeyPatch,
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    """Phase 142, task 01 repro, flipped green by task 02 (the strict
+    xfail marker is removed — a pass is now a pass).
+
+    The owner's KB shape: one source whose OLDER documents sort
+    lexicographically FIRST (``old/*``) and whose NEWER documents sort
+    LAST (``zz_new/*``). With ``BOR_OVERVIEW_INPUT_MAX_CHARS`` forced to
+    150, the 6-doc list (275 chars) exceeds the cap and the cut keeps a
+    PREFIX. Pre-fix that prefix was the lexicographic one — the three
+    OLDER ``old/*`` docs — so the stored outline's 8-token digest named
+    the old docs (``mydocs old 01 md old 01 summary of``) and the newer
+    ``zz_new/*`` docs never reached the model (the "summaries only up to
+    phase 113" failure at fixture scale). POST-fix the ``created_at``-
+    desc ordering (task 02) puts the newest docs first, so they survive
+    the cap and are the ones the outline names
+    (``mydocs zz new 03 md zz new 03``).
+    """
+    llm = OverviewDigestEmbedder()
+    get_settings.cache_clear()
+    monkeypatch.setenv("BOR_OVERVIEW_INPUT_MAX_CHARS", "150")
+    try:
+        rc, out = _run_main(monkeypatch, llm, ["--source", str(owner_kb)], capsys)
+    finally:
+        get_settings.cache_clear()
+    assert rc == 0
+    assert "added=6" in out
+    # 6 doc summaries (phase 118) + the overview + 3 folder summaries
+    # (source root, old/, zz_new/) — the overview gate fired (added > 0).
+    assert "overview=updated" in out
+    assert len(llm.chat_calls) == 10
+    overview_call = next(
+        m for m in llm.chat_calls
+        if "KB_OVERVIEW_MODE" in m[0]["content"]
+    )
+    user = next(m["content"] for m in overview_call if m["role"] == "user")
+    assert "[…truncated…]" in user  # the 150-char cap bit (the repro's precondition)
+
+    row = _row(db)
+    assert row is not None
+    # POST-fix digest: the capped list leads with the NEWEST docs —
+    # "MyDocs — zz_new/03.md — Zz New 03 — Summary of #" → first 8
+    # tokens "mydocs zz new 03 md zz new 03". Pre-fix the stored row
+    # would have been "mydocs old 01 md old 01 summary of" (the oldest
+    # batch) — the "summaries only up to phase 113" shape this pin
+    # forbids.
+    assert row.content == "Knowledge base outline:\n- mydocs zz new 03 md zz new 03"
+    assert "old" not in row.content  # no oldest-batch token may survive the cut