phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s

Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-13 19:28:05 -04:00
parent cec819743d
commit ee3efb28c9
113 changed files with 8228 additions and 344 deletions
+60 -36
View File
@@ -70,9 +70,11 @@ task 04):
(the count is the subfolder's recursive subtree — every document
whose path equals the folder or starts with ``folder + "/"``, the
same set the sync-time folder summary describes — and the file
lines ``source: X | path: Y | title: Z`` (the canonical
``read``/``grep`` identity — the phase-63 labeled format,
unchanged) in path order (``GET /api/docs`` order), capped at
lines ``source: X | path: Y | title: Z | date: YYYY-MM-DD`` (the
canonical ``read``/``grep`` identity — the phase-63 labeled format —
plus the phase-106 D5 ``date`` field APPENDED after ``title``; only
FILE lines carry a date — source/folder lines are not documents)
in path order (``GET /api/docs`` order), capped at
:data:`LS_MAX_FILE_LINES` lines + one deterministic grep-pointer
note for the rest (a 500-file folder costs 50 lines, never 500);
a ``source/folder`` ``path``: that folder's subfolders + own file
@@ -463,7 +465,8 @@ NO_DOCUMENT_DID_YOU_MEAN_MANY = (
SUGGESTION_LIMIT = 3
#: The drill-down ``ls`` file-line cap (phase 94, task 03): a folder's
#: own files list at most this many ``source: X | path: Y | title: Z``
#: own files list at most this many
#: ``source: X | path: Y | title: Z | date: YYYY-MM-DD``
#: lines (path order), then one deterministic grep-pointer note — a
#: 500-file folder costs the model 50 lines + the note, never 500.
#: Pinned module constant (no env var — the phase-94 TODO asks for a
@@ -655,14 +658,17 @@ def _source_root_summaries(db: Session) -> list[tuple[str, str]]:
]
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str]]:
"""``(path, title)`` of every document under *source*, ordered by
``path`` — the one bounded fetch a folder drill level lists (phase
94 task 03; one source's paths, not the whole KB)."""
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str, str]]:
"""``(path, title, created_iso_date)`` of every document under
*source*, ordered by ``path`` — the one bounded fetch a folder
drill level lists (phase 94 task 03; one source's paths, not the
whole KB). The date is the row's ``created_at`` UTC date part
(``YYYY-MM-DD``, phase 106 D5 — the ``ls`` FILE line's appended
`` | date: …`` field; only file lines carry a date)."""
return [
(path, title)
for path, title in db.execute(
select(Document.path, Document.title)
(path, title, created_at.strftime("%Y-%m-%d"))
for path, title, created_at in db.execute(
select(Document.path, Document.title, Document.created_at)
.where(Document.source == source)
.order_by(Document.path)
)
@@ -707,15 +713,17 @@ def ls_top(db: Session) -> list[tuple[str, int, str | None]]:
def group_folder_listing(
source: str,
folder: str,
rows: Sequence[tuple[str, str]],
rows: Sequence[tuple[str, str, str]],
summaries: Mapping[str, str],
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
) -> tuple[
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
]:
"""One level of the drill-down tree (phase 94, task 03) — pure.
Given *rows* — the source's ``(path, title)`` pairs in catalog
(path) order — and *summaries* (the source's stored
``folder_summaries`` rows: ``folder_path → summary``), the folder
level *folder* (source-relative; ``""`` = the source root):
Given *rows* — the source's ``(path, title, created_iso_date)``
triples in catalog (path) order — and *summaries* (the source's
stored ``folder_summaries`` rows: ``folder_path → summary``), the
folder level *folder* (source-relative; ``""`` = the source root):
* **(a) direct subfolders** — the folders whose parent is exactly
*folder*, in path order, each
@@ -732,8 +740,10 @@ def group_folder_listing(
prefix before the last ``/`` —
:func:`app.rag.folder_summaries.folder_of`, the shared notion) IS
*folder*, in path order (catalog order — the same order
``GET /api/docs`` serves), as ``(source, path, title)`` triples
— the canonical ``read``/``grep`` identity, capped at
``GET /api/docs`` serves), as ``(source, path, title, date)``
4-tuples — the canonical ``read``/``grep`` identity plus the
phase-106 D5 ``date`` field (the row's ``created_at`` UTC date
part, APPENDED — never inserted before ``title``), capped at
:data:`LS_MAX_FILE_LINES` (the rest fold into the renderer's
note; a 500-file folder never costs 500 lines).
* **(c) the TOTAL direct-file count** — pre-cap, for the note.
@@ -745,7 +755,7 @@ def group_folder_listing(
# indexed path (the existence rule's candidate set — a folder is
# present iff at least one path starts with ``folder + "/"``).
folders: set[str] = set()
for path, _title in rows:
for path, _title, _date in rows:
f = folder_of(path)
while f:
folders.add(f)
@@ -755,7 +765,7 @@ def group_folder_listing(
# folder + "/"`` arm (the folder's true descendants), one pass per
# document.
counts: dict[str, int] = {f: 0 for f in folders}
for path, _title in rows:
for path, _title, _date in rows:
if path in folders:
counts[path] += 1
f = folder_of(path)
@@ -767,8 +777,8 @@ def group_folder_listing(
for g in sorted(g for g in folders if folder_of(g) == folder)
]
files = [
(source, path, title)
for path, title in rows
(source, path, title, date)
for path, title, date in rows
if folder_of(path) == folder
]
return subfolders, files[:LS_MAX_FILE_LINES], len(files)
@@ -776,7 +786,9 @@ def group_folder_listing(
def ls_folder(
db: Session, source: str, folder: str
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
) -> tuple[
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
]:
"""One folder level of the drill-down ``ls`` (phase 94, task 03).
The source's document rows (:func:`_source_document_rows`) and
@@ -793,7 +805,7 @@ def ls_folder(
)
def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
def _folder_exists_in(rows: Sequence[tuple[str, str, str]], folder: str) -> bool:
"""The phase-94 folder-existence rule (``00_phase.md``), pure.
Folder *folder* (source-relative) under a registered source
@@ -805,11 +817,11 @@ def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
if not folder:
return True
prefix = folder + "/"
return any(path.startswith(prefix) for path, _title in rows)
return any(path.startswith(prefix) for path, _title, _date in rows)
def _deepest_existing_ancestor(
rows: Sequence[tuple[str, str]], folder: str
rows: Sequence[tuple[str, str, str]], folder: str
) -> str:
"""The deepest EXISTING folder prefix of a missing *folder* (pure).
@@ -854,7 +866,7 @@ def render_ls_top(entries: Sequence[tuple[str, int, str | None]]) -> str:
def render_folder_listing(
identity: str,
subfolders: Sequence[tuple[str, int, str | None]],
files: Sequence[tuple[str, str, str]],
files: Sequence[tuple[str, str, str, str]],
total_files: int,
) -> str:
"""One folder level of the drill-down ``ls`` (phase 94, task 03) —
@@ -867,9 +879,10 @@ def render_folder_listing(
below the header — a blank line, the 2-space-indented subfolder
lines `` {sub}/ — {m} documents`` in path order (``: {summary}``
appended ONLY when the subfolder's summary is stored), a blank
line, the file lines in EXACTLY the existing
``source: X | path: Y | title: Z`` format (the canonical
``read``/``grep`` identity — unchanged), and the cap note
line, the file lines in EXACTLY the
``source: X | path: Y | title: Z | date: YYYY-MM-DD`` format (the
canonical ``read``/``grep`` identity plus the phase-106 D5
appended ``date`` field — the only changed part), and the cap note
``…and {hidden} more documents in this folder — use grep
(pattern) to find a specific one.`` ONLY when the folder's own
files outnumber :data:`LS_MAX_FILE_LINES` (*files* arrives capped;
@@ -891,8 +904,8 @@ def render_folder_listing(
if files or total_files > len(files):
body.append("")
body.extend(
f"source: {source} | path: {path} | title: {title}"
for source, path, title in files
f"source: {source} | path: {path} | title: {title} | date: {date}"
for source, path, title, date in files
)
hidden = total_files - len(files)
if hidden > 0:
@@ -1172,15 +1185,26 @@ def _execute_tool(
holder.read_truncations.append(
(cast("str", raw_path), cap, len(doc.content))
)
# Phase 106 (D5): the date rides every document the model
# sees — the ``read`` result's SECOND line; the FIRST line
# stays ``Document {source}/{path}:`` BYTE-IDENTICAL (the
# E2E mock's ``_READ_RESULT_PREFIX`` header contract).
return (
f"Document {doc.source}/{doc.path}:\n"
f"date: {doc.created_at:%Y-%m-%d}\n"
f"{doc.content[:cap]}\n"
f"{TRUNCATION_MARKER}\n"
f"{READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))}"
)
# At or under the cap: byte-identical to the pre-phase-95 result
# (no marker, no notice, no holder entry, no ToolResultPiece).
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
# At or under the cap: the pre-phase-95 result plus the
# phase-106 D5 date line (first line byte-identical — the
# mock's header contract; no marker, no notice, no holder
# entry, no ToolResultPiece).
return (
f"Document {doc.source}/{doc.path}:\n"
f"date: {doc.created_at:%Y-%m-%d}\n"
f"{doc.content}"
)
if call.name == "grep":
raw_pattern = call.arguments.get("pattern")
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
+26
View File
@@ -18,6 +18,13 @@ The guarantees (phase 49 locked decisions):
unpack directory, and device/FIFO members — and counting every
extracted byte against a cap (zip-bomb guard). Any failure removes the
partial ``target_dir`` so no half-unpacked tree survives.
* :func:`unpack_archive` also restores each regular file's member mtime
(the zip DOS ``date_time`` or the tar ``mtime``) — phase 106, D2:
uploaded archives keep their file dates, which the importer reads as
the document creation date. Directories, symlinks, and hardlinks are
untouched; the cap and every safety check above are unchanged (the
``utime`` sits after a successful ``_write_capped``, so a failed
unpack still removes the partial tree).
* :func:`swap_in` makes ``new_dir`` become ``final_dir`` with **no
missing window**: the previous folder is renamed to a unique
same-filesystem ``.old-`` sibling first, the new folder is renamed
@@ -33,6 +40,7 @@ import stat
import tarfile
import uuid
import zipfile
from datetime import UTC, datetime
from pathlib import Path, PurePosixPath
from typing import IO
@@ -183,6 +191,16 @@ def _unpack_zip(archive: Path, target: Path, max_extract_bytes: int) -> None:
else:
with zf.open(member) as src:
_write_capped(src, dest, max_extract_bytes, total)
# Phase 106 (D2): restore the member's DOS mtime — a
# tz-agnostic epoch value, UTC-rendered exactly like an
# mtime — so uploaded archives keep their file dates.
# Regular files only; an OSError here still removes the
# partial tree in unpack_archive like any write failure.
# ``ns=`` takes INTEGER nanoseconds (a float seconds
# value raises TypeError), so convert explicitly.
mtime = datetime(*member.date_time, tzinfo=UTC).timestamp()
mtime_ns = int(mtime * 1_000_000_000)
os.utime(dest, ns=(mtime_ns, mtime_ns))
def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
@@ -206,6 +224,14 @@ def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
if src is None:
raise ArchiveUploadError("corrupt archive member")
_write_capped(src, dest, max_extract_bytes, total)
# Phase 106 (D2): restore the member's mtime (epoch
# seconds — a tz-agnostic value) so uploaded archives
# keep their file dates. Regular files only; an OSError
# here still removes the partial tree in unpack_archive.
# ``ns=`` takes INTEGER nanoseconds (a float seconds
# value raises TypeError), so convert explicitly.
mtime_ns = int(member.mtime * 1_000_000_000)
os.utime(dest, ns=(mtime_ns, mtime_ns))
else: # char/block device, FIFO
raise ArchiveUploadError(
"tar archives with device or FIFO members are not allowed"
+65
View File
@@ -0,0 +1,65 @@
"""Document-creation-date sourcing + normalization (phase 106, D2/D3).
Every document date the importer writes and every date the owner
edits passes through :func:`normalize_doc_date` — the single choke
point for the owner's rules: an UNDETERMINED date (no source signal)
and a FUTURE date (beyond a small clock-skew tolerance) both assume
the document was created TODAY (UTC). Naive source timestamps (zip
DOS mtimes, tar mtimes, git-free fallbacks) are tz-agnostic epoch-
based values rendered as UTC; aware ones are converted to UTC.
Pure and stdlib-only by contract (unit-pinned in
``tests/unit/test_doc_dates.py``): no database, no logging, no I/O
besides :func:`file_mtime_datetime`'s single ``stat`` — the callers
(importer task 04, the date-edit API task 05) own everything else.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from pathlib import Path
__all__ = ["FUTURE_SKEW_TOLERANCE", "file_mtime_datetime", "normalize_doc_date"]
#: Clock-skew tolerance (D3): a source date up to this far in the
#: FUTURE is a drifting clock, not a future document — it keeps its
#: date. Beyond it, the owner's rule applies (→ today).
FUTURE_SKEW_TOLERANCE = timedelta(days=1)
def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime:
"""*raw* → the stored UTC creation date (the D3 rule, pinned).
``now`` is injectable (tests); it defaults to
``datetime.now(UTC)``. ``raw=None`` (undetermined) → *now*;
naive *raw* → treated as UTC; aware *raw* → converted to UTC;
*raw* beyond *now* + :data:`FUTURE_SKEW_TOLERANCE` → *now*.
The result always carries full precision (no date-truncation —
the display formats, the storage doesn't).
"""
if now is None:
now = datetime.now(UTC)
elif now.tzinfo is None:
# The future check compares in AWARE space — a naive ``now``
# (callers/tests) is a UTC instant, like the naive ``raw``.
now = now.replace(tzinfo=UTC)
if raw is None:
return now
# Epoch-based source values (zip DOS times, tar mtimes) are
# tz-agnostic — attach UTC; never assume the host's local TZ.
# Aware values are converted to UTC (the comparison below is
# done in aware space).
raw = raw.replace(tzinfo=UTC) if raw.tzinfo is None else raw.astimezone(UTC)
if raw > now + FUTURE_SKEW_TOLERANCE:
# Genuinely future (beyond the clock-skew tolerance) → today.
return now
return raw
def file_mtime_datetime(path: Path) -> datetime:
"""The file's mtime as an aware UTC datetime (the D2 fallback).
Epoch mtimes are tz-agnostic — UTC is the correct rendering
(zip DOS timestamps and tar mtimes pass through the same
:func:`normalize_doc_date` after unpacking, task 03).
"""
return datetime.fromtimestamp(path.stat().st_mtime, tz=UTC)
+80 -3
View File
@@ -31,6 +31,17 @@ previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
summary line with per-format counts (PLAN §9).
Document dates (phase 106, D2/D4): every import sources
``documents.created_at`` from the file's source — the per-file git
last-commit date when a ``doc_dates_by_root`` entry names the file,
else the file's mtime — normalized by
:func:`app.rag.doc_dates.normalize_doc_date` (undetermined or future →
today, D3) on every add and update. On the unchanged path the stored
date is REFRESHED from the same source (it may go OLDER — no monotonic
guard) and counted in ``summary.dates_updated`` — unless the row
carries the owner's manual correction (``created_at_manual``, D1), which
the sync never touches.
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
"""
@@ -51,6 +62,7 @@ from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.rag.chunker import chunk_document, extract_title
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
from app.rag.llm import EmbeddingError, LLMError
from app.rag.summarizer import generate_summary
@@ -93,6 +105,12 @@ class ImportSummary:
#: Non-markdown files whose summary generation failed (best-effort —
#: the document is still indexed, without a summary).
summary_errors: int = 0
#: Files whose ``created_at`` was refreshed on the UNCHANGED path —
#: content untouched, date re-sourced (phase 106, D4: the date may
#: go OLDER; a date-only refresh NEVER counts added/updated/pruned,
#: so no ``sources_meta`` bump, no overview/folder-summary
#: regeneration).
dates_updated: int = 0
#: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
formats: dict[str, int] = field(default_factory=dict)
@@ -107,7 +125,7 @@ class ImportSummary:
logger.info(
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"formats=%s",
"dates_updated=%d formats=%s",
self.files,
self.added,
self.updated,
@@ -118,6 +136,7 @@ class ImportSummary:
self.embed_batches,
self.summaries,
self.summary_errors,
self.dates_updated,
self.format_counts(),
)
@@ -248,6 +267,7 @@ async def import_sources(
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring).
@@ -288,6 +308,19 @@ async def import_sources(
flag ON and is walked again with it OFF simply never enters
``seen``, so the next ``prune=True`` run deletes its row
automatically (A2 — the A9/phase-89 precedent).
``doc_dates_by_root`` (phase 106, D2/D4) maps ``str(root)`` — the
root path string exactly as passed in *sources* — to that source's
RAW per-file source dates: source-relative POSIX path → the git
last-commit datetime (task 03's ``file_commit_dates``). ONLY git
roots are listed — unlisted roots (local dirs, unpacked uploads)
take the mtime fallback, and a path missing from its root's map
does too. The map entry beats the file's mtime when present. The
progress pre-walk is untouched (dates change no file count).
``None`` (the default) changes nothing for existing callers: the
mtime fallback applies to every file — which IS the behavior
change, D4: an unchanged file now refreshes its stored date from
its source on every run (the backfill-correction case).
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -325,6 +358,10 @@ async def import_sources(
source_names.add(source)
ignore = _ignore_for_root(root, ignore_by_root)
include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
# Phase 106 (D2): the root's raw source dates (git last-commit
# for git roots, keyed by the same str(root) convention); {}
# for unlisted roots — every file then takes the mtime fallback.
dates_map = (doc_dates_by_root or {}).get(str(root), {})
for path in iter_importable_files(
root,
llm.settings.import_extension_set,
@@ -353,7 +390,7 @@ async def import_sources(
try:
await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm,
summary=summary,
summary=summary, raw_date=dates_map.get(rel),
)
except EmbeddingError as e:
# A pathological file (e.g. content the embedding endpoint
@@ -384,15 +421,45 @@ async def _index_file(
full_path: Path,
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
) -> None:
"""Upsert one file: doc row + chunk rows + embeddings, one transaction."""
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
``raw_date`` (phase 106, D2) is the file's RAW source date — the
git last-commit datetime from the caller's ``doc_dates_by_root``
map, or ``None`` (every non-git case): the file's mtime is read
here, once, and becomes the source date (the D2 fallback).
"""
settings = llm.settings
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
if raw_date is None:
# D2 fallback: no source date in the map → the file's mtime
# (one stat). Read before the unchanged early-return — the
# unchanged path refreshes the stored date from the same source.
raw_date = file_mtime_datetime(full_path)
if doc is not None and doc.content_hash == digest:
summary.unchanged += 1
logger.info("import: unchanged source=%s path=%s", source, rel)
if doc.created_at_manual:
# D1/D4: the owner's correction survives the sync — no
# write at all (the phase-97 ``manually_edited`` precedent).
return
# D4: the date refreshes on every sync, including unchanged
# files, and may go OLDER (no monotonic guard). A date-only
# refresh is still counted ``unchanged`` — never added/updated/
# pruned, so no ``sources_meta`` bump and no regeneration.
target = normalize_doc_date(raw_date)
if target != doc.created_at:
doc.created_at = target
session.commit()
summary.dates_updated += 1
logger.info(
"import: date-refreshed source=%s path=%s date=%s",
source, rel,
doc.created_at.isoformat(),
)
return
verb = "updated" if doc is not None else "added"
@@ -411,6 +478,11 @@ async def _index_file(
content=content,
content_hash=digest,
indexed_at=datetime.now(UTC),
# Phase 106 (D2/D3): the sourced creation date, normalized
# (undetermined or future → today). ``created_at_manual``
# stays the column default (False) — only the date-edit API
# (task 05) sets it.
created_at=normalize_doc_date(raw_date),
)
session.add(doc)
else:
@@ -419,6 +491,11 @@ async def _index_file(
doc.content = content
doc.content_hash = digest
doc.indexed_at = datetime.now(UTC)
# Phase 106 (D4): a content change is a new document version —
# the date is re-sourced and a previous manual correction is
# reset (it referred to the old content).
doc.created_at = normalize_doc_date(raw_date)
doc.created_at_manual = False
session.flush() # guarantees doc.id even for brand-new rows
+12 -1
View File
@@ -352,6 +352,12 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
cap — not the prompt — decides whether the tools are actually
offered to the model, see :mod:`app.rag.agent`).
Each ``<document>`` block carries the identity attributes
``source`` / ``path`` / ``title`` — and, since phase 106 (D5),
``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``,
APPENDED after ``title`` — the only position; always present,
``created_at`` is NOT NULL) — plus the document's full text.
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
LEADING this section (the document texts are already context — do
not ``read`` one the user asked to open) was tried and REVERTED:
@@ -368,8 +374,13 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
# attribute at the ``source``/``path`` copy site was TRIED and
# REVERTED the same day (no improvement across runs; the block stays
# exactly the document identity + full text).
# Phase 106 (D5): every document the model sees carries its
# creation date — the block's ``date`` attribute (the row's
# ``created_at`` UTC date part, appended after ``title`` — the
# only position; always present, ``created_at`` is NOT NULL).
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}">\n'
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
f'date="{doc.created_at:%Y-%m-%d}">\n'
f"{doc.content}\n"
"</document>"
for doc in documents
+81 -1
View File
@@ -41,10 +41,12 @@ Deterministic tie-break for equal fused scores:
"""
from __future__ import annotations
import math
import re
import uuid
from collections.abc import Sequence
from dataclasses import dataclass, replace
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import select, text
@@ -146,6 +148,7 @@ _LEXICAL_SQL = text(
d.content AS doc_content,
d.content_hash AS content_hash,
d.indexed_at AS indexed_at,
d.created_at AS created_at,
c.is_summary AS is_summary,
ts_rank(c.tsv, to_tsquery('english', :tsquery)) AS rank
FROM chunks c
@@ -242,6 +245,73 @@ def fuse(
return out
def apply_recency_boost(
chunks: Sequence[RetrievedChunk],
*,
now: datetime | None = None,
weight: float | None = None,
half_life_days: int | None = None,
) -> list[RetrievedChunk]:
"""Additive recency boost on the fused score (phase 106, D6).
Each chunk's score becomes
``score + weight * exp(−age_days / half_life_days)`` where
``age_days = max(0, (now − document.created_at))`` in days — a
zero-age document gets the full *weight* (the MAXIMUM additive
score), each ``half_life_days`` of age multiplies the remaining
boost by ``e**-1`` (≈0.37), and a FUTURE ``created_at`` clamps to
age 0 (the document reads as brand-new — consistent with D3's
today-folding in :mod:`app.rag.doc_dates`). Defaults: *weight* /
*half_life_days* from :func:`get_settings` (``recency_boost`` /
``recency_half_life_days``), *now* from ``datetime.now(UTC)``.
Magnitude rationale (the ``0.0007`` default, the k=60 RRF scale):
rank 1 vs 2 in one list differs by ~0.00026 and rank 1 vs 10 by
~0.0021, so the full weight is a bounded 2-3 rank head start —
enough to break near-ties toward the newer document, far below the
fused gap between a document that answers and one that merely
resembles (the phase-106 fine-line battery pins the measured
margin: 0.00263 ≥ 3× the zero-age boost).
Pure (the :func:`fuse` convention): the inputs are never mutated —
every boosted chunk is a ``replace()`` copy — and the result is
re-sorted with the EXISTING deterministic key
``(−score, −cosine, document.path, position)``; with ``weight=0``
every score is untouched and an already-fused (already-sorted)
input comes back byte-identical (the kill switch, pinned).
Untouched by design: the A8 honesty gate and ``query_log.top_score``
(both read the chunk's ``cosine``, which the boost never modifies),
:func:`weak_hit_titles` (titles only), and the never-truncated
top-N contract (:func:`select_documents` still feeds whole
documents — the boost re-ranks WHICH documents, never truncates).
SINGLE APPLY SITE: :func:`retrieve()` is the only caller in
``app/`` — the chat API and ``scripts/eval_retrieval.py`` inherit
the boost through it; nothing else may apply it.
"""
if weight is None or half_life_days is None:
settings = get_settings()
if weight is None:
weight = settings.recency_boost
if half_life_days is None:
half_life_days = settings.recency_half_life_days
if half_life_days <= 0:
raise ValueError("half_life_days must be > 0")
if now is None:
now = datetime.now(UTC)
boosted: list[RetrievedChunk] = []
for rc in chunks:
age_days = max(0.0, (now - rc.document.created_at).total_seconds() / 86400.0)
boosted.append(
replace(
rc,
score=rc.score + weight * math.exp(-age_days / half_life_days),
)
)
boosted.sort(key=lambda rc: (-rc.score, -rc.cosine, rc.document.path, rc.position))
return boosted
def _vector_candidates(
db: Session, question_embedding: list[float], limit: int
) -> list[RetrievedChunk]:
@@ -289,6 +359,7 @@ _NAME_HIT_SQL = text(
d.content AS doc_content,
d.content_hash AS content_hash,
d.indexed_at AS indexed_at,
d.created_at AS created_at,
c.id AS chunk_id,
c.position AS position,
c.content AS content,
@@ -364,6 +435,7 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]:
content=row.doc_content,
content_hash=row.content_hash,
indexed_at=row.indexed_at,
created_at=row.created_at,
)
out.append(
RetrievedChunk(
@@ -418,6 +490,7 @@ def _lexical_candidates(db: Session, question: str, limit: int) -> list[Retrieve
content=row.doc_content,
content_hash=row.content_hash,
indexed_at=row.indexed_at,
created_at=row.created_at,
)
out.append(
RetrievedChunk(
@@ -460,7 +533,14 @@ def retrieve(
raise ValueError("lexical_candidates must be >= 1")
vector = _vector_candidates(db, question_embedding, v_n)
lexical = _lexical_candidates(db, question, l_n)
return fuse(vector, lexical, settings.rrf_k)
fused = fuse(vector, lexical, settings.rrf_k)
if settings.recency_boost > 0:
# Phase 106, D6 — the SINGLE recency-boost apply site: an
# additive post-fusion re-rank (see :func:`apply_recency_boost`).
# ``0`` = off: the pre-phase ranking returns byte-identical (the
# kill switch) and weight-0 callers pay nothing.
return apply_recency_boost(fused)
return fused
def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]: