phase: 106_document_dates

Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.

.agents/phases/complete/106_document_dates/07_recency_boost.md

@@ -0,0 +1,61 @@
+# Task 07 — Recency boost: newer documents rank higher, without breaking retrieval (D6)
+
+**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Newer documents should be ranked higher in retrieval somehow, or at least given a boost, without breaking the existing retrieval process (so make sure to test with documents that have the correct answer but are older against documents that are similar and newer but don't quit correctly answer the question). This will be a fine line to walk, so testing is crucial here."
+
+## Objective
+A small, env-tunable, kill-switchable additive recency term on the RRF-fused score — applied once in `retrieve()` after `fuse()` — so a fresh document gets a bounded head start on near-ties while an older document that ACTUALLY answers the question keeps its rank. The owner's scenario is pinned by a permanent real-Postgres battery with deterministic axis vectors. The A7 math, the A8 cosine gate, `query_log.top_score`, and the never-truncated contract are untouched.
+
+## Work
+1. `app/config.py` — the hybrid block (L163-174, after `rrf_k`):
+   ```python
+   #: Recency boost on the RRF-fused retrieval score (phase 106, D6): the
+   #: MAXIMUM additive score a zero-age document gets —
+   #: ``fused + recency_boost * exp(-age_days / recency_half_life_days)``
+   #: (``app.rag.retriever.apply_recency_boost``, applied in
+   #: ``retrieve()`` after ``fuse()``). ``0`` = off — the pre-phase
+   #: ranking is byte-identical (the kill switch); negative values fail
+   #: startup loudly (the ``agent_max_rounds`` validator pattern).
+   #: 0.001 ≈ a 1-2 rank head start on a 60+ RRF scale — enough to break
+   #: near-ties toward the newer document, far below the gap between a
+   #: document that answers and one that merely resembles (the
+   #: phase-106 fine-line battery pins it).
+   recency_boost: float = 0.001
+   #: Age (days) at which the recency boost halves (phase 106, D6).
+   #: ``<= 0`` fails startup loudly (same validator family).
+   recency_half_life_days: int = 365
+   ```
+   Add the startup validator (find `agent_max_rounds`'s field-validator and follow it — fail loudly naming the field): `recency_boost < 0` → error; `recency_half_life_days <= 0` → error. `.env.example` — document `BOR_RECENCY_BOOST` + `BOR_RECENCY_HALF_LIFE_DAYS` (the hybrid section, the existing comment style).
+2. `app/rag/retriever.py`:
+   - NEW pure function (module-level, next to `fuse`):
+     ```python
+     def apply_recency_boost(
+         chunks: Sequence[RetrievedChunk],
+         *,
+         now: datetime | None = None,
+         weight: float | None = None,
+         half_life_days: int | None = None,
+     ) -> list[RetrievedChunk]:
+     ```
+     Defaults from `get_settings()` when omitted; `now` defaults to `datetime.now(UTC)`. For each chunk: `age_days = max(0.0, (now − doc.created_at).total_seconds() / 86400.0)` (a future `created_at` clamps to 0 — consistent with D3's today-folding), `score = score + weight * math.exp(−age_days / half_life_days)` (import `math`; `replace(rc, score=new_score)` — never mutate inputs, the `fuse` convention). Return the list re-sorted with the EXISTING deterministic key `(−score, −cosine, document.path, position)` — with `weight=0` every score is untouched and the order is byte-identical (pinned). Docstring: the D6 contract, the magnitude rationale (0.001 ≈ 1-2 ranks on the k=60 scale — rank 1 vs 2 in one list differs by ~0.00026, rank 1 vs 10 by ~0.0021), the untouched surfaces (A8 gate = cosine, `query_log.top_score` = cosine, `weak_hit_titles` = titles only, the never-truncated top-N), and the single-apply-site rule (`retrieve()` only — chat API + `eval_retrieval` inherit it).
+   - `retrieve()` (L398-425): after `return fuse(vector, lexical, settings.rrf_k)` → apply: `fused = fuse(...)`; `if settings.recency_boost > 0: return apply_recency_boost(fused)`; `return fused` (weight-0 callers pay nothing).
+3. `scripts/eval_retrieval.py` — the printed top-N table gains two columns: the document's `created_at` (UTC date) and the post-boost effective score (the script calls `retrieve()`, which now applies the boost — print both the raw fused and effective where they differ, or just effective + date; keep the verdict column). Docstring line updated.
+4. Tests:
+   - `tests/unit/test_retriever_recency.py` (NEW — fake rows, no DB): age 0 → `+weight` exact; age = half-life → `+weight*exp(-1)` (±1e-9); age 10× half-life → ~`+weight*exp(-10)` (assert `< weight * 1e-3`); future date → full weight (the clamp); `weight=0` → the returned list's `(score, order)` is byte-identical to the input (the kill-switch pin); a tie on raw score breaks toward the newer document; the sort key's `(path, position)` tie-break still applies when scores AND cosines are equal (two docs, same age).
+   - `tests/integration/test_recency_boost.py` (NEW — real Postgres, `tests/integration/test_name_hit_lexical.py`'s axis-vector idiom VERBATIM: `D=768` unit vectors, exact cosines, `TRUNCATE chunks, documents` fixture, `retrieve()` + `select_documents()` with settings overrides via the house settings-override pattern — check how that suite's siblings inject settings, e.g. `monkeypatch` on `get_settings` or `Settings(_env_file=None, …)`):
+     1. **THE OWNER SCENARIO (old-correct beats new-similar).** Question `"How did I configure the backup retention policy?"`. Doc A `backups/retention.md`, `created_at=2020-01-01`: the exact answer — chunk vector = the question vector's axis (cosine 1.0) + its exact tokens in the chunk text (top FTS rank). Doc B `backups/retention-draft.md`, `created_at=yesterday` (the test computes `now − 1d`): topically similar (shares `backup retention policy` tokens — a solid FTS hit at rank 2-3) but a weaker vector (half-parallel axis → cosine ~0.707) and its text says the policy is "under review, no decision yet" (no answer). Assert with DEFAULTS: `select_documents(...)[0].path == "backups/retention.md"` AND the fused (pre-boost, computed via `fuse` directly in the test for the margin) gap A−B ≥ 3× the zero-age boost (record the measured margin in the test docstring — the "comfortable margin" requirement). Assert AGAIN with `recency_boost=0` (settings override): A still first (no-regression pin — relevance alone ordered them).
+     2. **The boost is real (near-tie flips toward newer).** Docs C (2019) and D (yesterday) with IDENTICAL chunk text + IDENTICAL vectors (a true tie — same fused score, cosine, FTS rank; the deterministic sort key would otherwise order by path, and path is set so the OLDER sorts first lexicographically, e.g. `c-older.md` < `d-newer.md`). With defaults: D (newer) is first. With `weight=0`: C (older) is first (proving the boost — not drift — is the differentiator).
+     3. **Decay end-to-end:** the same C/D pair with D aged to `half_life + 365` days (≈ `weight*e^{-3}` ≈ 0.00005, below the tie gap 0) → C first again (the boost faded — recency is an age signal, not a binary).
+     4. **The gate is untouched:** the owner-scenario question's `max cosine` (the A8 input) equals the pre-boost run's (assert on the retrieved chunks' `cosine` values — the boost never touches them).
+   - If test 1's measured margin under the DEFAULTS is thin (< 3× the boost) or the scenario flips, tune the DEFAULTS (0.001/365 are the starting point — the owner re-tunes live via the env) until old-correct wins comfortably, and record the final margin in the docstring. The test asserts the SEMANTICS (A first, margin ≥ 3× boost), never the exact floats.
+5. Run `uv run pytest tests/unit/test_retriever_recency.py tests/integration/test_recency_boost.py -q` (DB up) — green; then `uv run pytest tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the retriever's existing contract suites stay green — the boost is ON by default in them, so any drift surfaces here).
+
+## Testing & Quality
+- Unit: the decay/weight/clamp/tie/kill-switch pins (pure function).
+- Integration: the fine-line battery on real Postgres with exact axis cosines — the owner's scenario + the near-tie flip + the decay + the cosine-gate-untouched pin.
+- Coverage: **>90%** on `app/` (config validator + retriever branches covered — the validate.sh gate).
+
+## Completion Criteria
+- [ ] `Settings.recency_boost` (default 0.001, 0 = byte-identical off, negative fails startup) + `recency_half_life_days` (default 365, `<= 0` fails startup); `.env.example` documents both
+- [ ] `apply_recency_boost` is pure (defaults from settings, `now` injectable, inputs unmutated, the existing 4-key sort) and is applied in `retrieve()` after `fuse()` and ONLY there — chat API + `eval_retrieval` inherit it; `eval_retrieval` prints the date + effective score
+- [ ] The owner's scenario is pinned: older-correct beats newer-similar under defaults (margin ≥ 3× the zero-age boost, recorded) AND with the boost off; the near-tie flips toward the newer with the boost on and back without; the decay pin holds; the A8 cosine input is untouched
+- [ ] `tests/unit/test_retriever_recency.py` + `tests/integration/test_recency_boost.py` + the two existing retriever-contract suites green; `uv run ruff check . && uv run pyright` clean