phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s

Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-13 19:28:05 -04:00
parent cec819743d
commit ee3efb28c9
113 changed files with 8228 additions and 344 deletions
+81 -1
View File
@@ -41,10 +41,12 @@ Deterministic tie-break for equal fused scores:
"""
from __future__ import annotations
import math
import re
import uuid
from collections.abc import Sequence
from dataclasses import dataclass, replace
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import select, text
@@ -146,6 +148,7 @@ _LEXICAL_SQL = text(
d.content AS doc_content,
d.content_hash AS content_hash,
d.indexed_at AS indexed_at,
d.created_at AS created_at,
c.is_summary AS is_summary,
ts_rank(c.tsv, to_tsquery('english', :tsquery)) AS rank
FROM chunks c
@@ -242,6 +245,73 @@ def fuse(
return out
def apply_recency_boost(
chunks: Sequence[RetrievedChunk],
*,
now: datetime | None = None,
weight: float | None = None,
half_life_days: int | None = None,
) -> list[RetrievedChunk]:
"""Additive recency boost on the fused score (phase 106, D6).
Each chunk's score becomes
``score + weight * exp(−age_days / half_life_days)`` where
``age_days = max(0, (now − document.created_at))`` in days — a
zero-age document gets the full *weight* (the MAXIMUM additive
score), each ``half_life_days`` of age multiplies the remaining
boost by ``e**-1`` (≈0.37), and a FUTURE ``created_at`` clamps to
age 0 (the document reads as brand-new — consistent with D3's
today-folding in :mod:`app.rag.doc_dates`). Defaults: *weight* /
*half_life_days* from :func:`get_settings` (``recency_boost`` /
``recency_half_life_days``), *now* from ``datetime.now(UTC)``.
Magnitude rationale (the ``0.0007`` default, the k=60 RRF scale):
rank 1 vs 2 in one list differs by ~0.00026 and rank 1 vs 10 by
~0.0021, so the full weight is a bounded 2-3 rank head start —
enough to break near-ties toward the newer document, far below the
fused gap between a document that answers and one that merely
resembles (the phase-106 fine-line battery pins the measured
margin: 0.00263 ≥ 3× the zero-age boost).
Pure (the :func:`fuse` convention): the inputs are never mutated —
every boosted chunk is a ``replace()`` copy — and the result is
re-sorted with the EXISTING deterministic key
``(−score, −cosine, document.path, position)``; with ``weight=0``
every score is untouched and an already-fused (already-sorted)
input comes back byte-identical (the kill switch, pinned).
Untouched by design: the A8 honesty gate and ``query_log.top_score``
(both read the chunk's ``cosine``, which the boost never modifies),
:func:`weak_hit_titles` (titles only), and the never-truncated
top-N contract (:func:`select_documents` still feeds whole
documents — the boost re-ranks WHICH documents, never truncates).
SINGLE APPLY SITE: :func:`retrieve()` is the only caller in
``app/`` — the chat API and ``scripts/eval_retrieval.py`` inherit
the boost through it; nothing else may apply it.
"""
if weight is None or half_life_days is None:
settings = get_settings()
if weight is None:
weight = settings.recency_boost
if half_life_days is None:
half_life_days = settings.recency_half_life_days
if half_life_days <= 0:
raise ValueError("half_life_days must be > 0")
if now is None:
now = datetime.now(UTC)
boosted: list[RetrievedChunk] = []
for rc in chunks:
age_days = max(0.0, (now - rc.document.created_at).total_seconds() / 86400.0)
boosted.append(
replace(
rc,
score=rc.score + weight * math.exp(-age_days / half_life_days),
)
)
boosted.sort(key=lambda rc: (-rc.score, -rc.cosine, rc.document.path, rc.position))
return boosted
def _vector_candidates(
db: Session, question_embedding: list[float], limit: int
) -> list[RetrievedChunk]:
@@ -289,6 +359,7 @@ _NAME_HIT_SQL = text(
d.content AS doc_content,
d.content_hash AS content_hash,
d.indexed_at AS indexed_at,
d.created_at AS created_at,
c.id AS chunk_id,
c.position AS position,
c.content AS content,
@@ -364,6 +435,7 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]:
content=row.doc_content,
content_hash=row.content_hash,
indexed_at=row.indexed_at,
created_at=row.created_at,
)
out.append(
RetrievedChunk(
@@ -418,6 +490,7 @@ def _lexical_candidates(db: Session, question: str, limit: int) -> list[Retrieve
content=row.doc_content,
content_hash=row.content_hash,
indexed_at=row.indexed_at,
created_at=row.created_at,
)
out.append(
RetrievedChunk(
@@ -460,7 +533,14 @@ def retrieve(
raise ValueError("lexical_candidates must be >= 1")
vector = _vector_candidates(db, question_embedding, v_n)
lexical = _lexical_candidates(db, question, l_n)
return fuse(vector, lexical, settings.rrf_k)
fused = fuse(vector, lexical, settings.rrf_k)
if settings.recency_boost > 0:
# Phase 106, D6 — the SINGLE recency-boost apply site: an
# additive post-fusion re-rank (see :func:`apply_recency_boost`).
# ``0`` = off: the pre-phase ranking returns byte-identical (the
# kill switch) and weight-0 callers pay nothing.
return apply_recency_boost(fused)
return fused
def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]: