phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s

Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-13 19:28:05 -04:00
parent cec819743d
commit ee3efb28c9
113 changed files with 8228 additions and 344 deletions
+185 -31
View File
@@ -10,6 +10,14 @@ update or clear ``documents.summary`` and re-embed the ``is_summary``
chunk (embed first, mutate second — a failed LLM call leaves the row and
chunk untouched; the content chunks are never re-embedded, D4).
PATCH /api/documents/date — the admin document-date editor (phase 106,
D7): set the owner's corrected ``documents.created_at`` (normalized
through ``app.rag.doc_dates.normalize_doc_date`` — a manually set future
date folds to today, D3) + flag it manual, or clear the manual flag
(null date — the stored date stands until the next sync). A pure DB
write: NO LLM/embedding call — a date is never embedded (the
deliberate contrast with the phase-57 ``is_summary`` re-embed).
GET /api/docs/tree — the admin's full recursive KB tree in one fetch
(phase 97, task 02): the same drill-down tree the agent's ``ls``
walks, with the file metadata the RAG view's rows and stat cards need
@@ -40,10 +48,13 @@ from app.core.auth import require_admin, require_user
from app.db import get_db
from app.models import Chunk, Document, FolderSummary
from app.rag.agent import list_source_names
from app.rag.doc_dates import normalize_doc_date
from app.rag.folder_summaries import MIN_DOCS_PER_FOLDER, folder_of
from app.rag.importer import match_extension
from app.rag.llm import EmbeddingError, LLMClient
from app.schemas import (
DateResult,
DateUpdate,
DocContent,
DocList,
DocSummary,
@@ -101,10 +112,18 @@ def list_indexed_documents(
Document.path,
Document.title,
func.count(Chunk.id).label("chunks"),
Document.created_at,
Document.indexed_at,
)
.outerjoin(Chunk, Chunk.document_id == Document.id)
.group_by(Document.id, Document.source, Document.path, Document.title, Document.indexed_at)
.group_by(
Document.id,
Document.source,
Document.path,
Document.title,
Document.created_at,
Document.indexed_at,
)
.order_by(Document.source, Document.path)
).all()
return DocList(
@@ -115,6 +134,7 @@ def list_indexed_documents(
path=row.path,
title=row.title,
chunks=row.chunks,
created_at=row.created_at.isoformat(),
indexed_at=row.indexed_at.isoformat(),
)
for row in rows
@@ -157,6 +177,7 @@ def get_document_content(
title=doc.title,
format=doc_format(doc.path, get_settings().import_extension_set),
summary=doc.summary,
created_at=doc.created_at.isoformat(),
content=doc.content,
indexed_at=doc.indexed_at.isoformat(),
chunks=chunks,
@@ -231,6 +252,75 @@ async def update_document_summary(
)
@router.patch("/documents/date", response_model=DateResult)
def update_document_date(
payload: DateUpdate,
db: Session = Depends(get_db), # noqa: B008
_admin: None = Depends(require_admin), # noqa: B008
) -> DateResult:
"""Set or clear the owner's correction of a document's creation date.
Admin-only (phase 106, D7 — the phase-57 split): the document
viewer itself stays user-gated (``/documents/content`` — admin OR
token holder); only this edit affordance is admin-gated. DB-only
(the ``/documents/content`` row-lookup rule): the ``(source, path)``
pair is looked up as a row — an unknown pair, including traversal
strings such as ``../../etc/passwd``, is simply not a row (→ 404
``document not found``), and there is no filesystem access. NO
LLM/embedding call — a date is never embedded (no chunk, no
retrieval role) — the deliberate contrast with the phase-57
``is_summary`` re-embed in :func:`update_document_summary`.
* **Set** (``date`` present) — ``datetime.fromisoformat`` accepts a
bare ``YYYY-MM-DD`` (midnight) and full ISO datetimes; a
MALFORMED value 422s here (the model field is an unconstrained
``str | None`` on purpose, so the detail can name the field). The
parse goes through
:func:`app.rag.doc_dates.normalize_doc_date` (D3 — the single
choke point: naive → UTC, aware → converted, a manually set
FUTURE date also folds to today — consistency with the sourced
path) and stores ``created_at`` + ``created_at_manual = True``
(the owner's correction — the sync-time importer then SKIPS the
refresh on this row, D1/D4).
* **Clear** (``date`` null/absent — the "revert to sync"
operation) — ``created_at_manual = False`` ONLY: the stored date
stands until the next sync refreshes it (the API cannot
re-read the source, D7).
The response echoes the stored state — ``created_at`` (ISO-8601)
+ ``created_at_manual`` — the viewer re-renders its Created badge
from it (no second fetch).
"""
doc = db.scalar(
select(Document).where(
Document.source == payload.source, Document.path == payload.path
)
)
if doc is None:
raise HTTPException(status_code=404, detail="document not found")
if payload.date:
try:
parsed = datetime.fromisoformat(payload.date)
except ValueError:
raise HTTPException(
status_code=422,
detail="date must be an ISO date or datetime (e.g. 2024-06-15)",
) from None
doc.created_at = normalize_doc_date(parsed)
doc.created_at_manual = True
else:
# The CLEAR (D7): drop the manual flag only — the stored date
# stands until the next sync refreshes it.
doc.created_at_manual = False
db.commit()
return DateResult(
source=doc.source,
path=doc.path,
created_at=doc.created_at.isoformat(),
created_at_manual=doc.created_at_manual,
)
@router.patch("/folders/summary", response_model=FolderSummaryResult)
def update_folder_summary(
payload: FolderSummaryUpdate,
@@ -305,16 +395,17 @@ def update_folder_summary(
#: One catalogue row the tree builder consumes:
#: ``(source, path, title, chunks, indexed_at)`` — the ``GET /api/docs``
#: query's columns minus the document ``id`` (the tree has no document
#: ids), in the same ``(source, path)`` order; ``indexed_at`` is the
#: ISO-8601 string the endpoint converts (the builder stays pure over
#: plain types — unit-testable without a DB).
TreeDocRow = tuple[str, str, str, int, str]
#: ``(source, path, title, chunks, indexed_at, created_at)`` — the
#: ``GET /api/docs`` query's columns minus the document ``id`` (the
#: tree has no document ids), in the same ``(source, path)`` order;
#: ``indexed_at`` / ``created_at`` are the ISO-8601 strings the
#: endpoint converts (the builder stays pure over plain types —
#: unit-testable without a DB).
TreeDocRow = tuple[str, str, str, int, str, str]
#: One of a source's file rows, already source-scoped:
#: ``(path, title, chunks, indexed_at)``.
TreeFileRow = tuple[str, str, int, str]
#: ``(path, title, chunks, indexed_at, created_at)``.
TreeFileRow = tuple[str, str, int, str, str]
def _folder_counts(
@@ -339,13 +430,13 @@ def _folder_counts(
own descendants, or the file wearing its name, exist).
"""
folders: set[str] = set()
for path, _title, _chunks, _indexed_at in rows:
for path, _title, _chunks, _indexed_at, _created_at in rows:
folder = folder_of(path)
while folder:
folders.add(folder)
folder = folder_of(folder)
counts: dict[str, int] = {folder: 0 for folder in folders}
for path, _title, _chunks, _indexed_at in rows:
for path, _title, _chunks, _indexed_at, _created_at in rows:
if path in folders:
counts[path] += 1
folder = folder_of(path)
@@ -381,27 +472,60 @@ def _level_children(
flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
stored ``folder_summaries`` row for ``(source, sub)`` — the same
rule the source node applies (see :func:`build_kb_tree`).
And, since phase 106 (D9), each folder node carries ``updated_at``
= the subtree's MAX document ``created_at``: the max over this
folder's direct files' dates and its subfolder children's (already
recursive) ``updated_at`` values, via :func:`_subtree_max`.
"""
children: list[KbTreeFolder | KbTreeFile] = []
for sub in sorted(g for g in folders if folder_of(g) == folder):
sub_children = _level_children(source, sub, folders, counts, rows, summaries)
children.append(
KbTreeFolder(
path=sub,
documents=counts[sub],
updated_at=_subtree_max(sub_children),
summary=summaries.get((source, sub)),
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
and (source, sub) not in summaries,
children=_level_children(source, sub, folders, counts, rows, summaries),
children=sub_children,
)
)
for path, title, chunks, indexed_at in rows:
for path, title, chunks, indexed_at, created_at in rows:
if folder_of(path) == folder:
children.append(
KbTreeFile(path=path, title=title, chunks=chunks, indexed_at=indexed_at)
KbTreeFile(
path=path,
title=title,
chunks=chunks,
created_at=created_at,
indexed_at=indexed_at,
)
)
return children
def _subtree_max(children: Sequence[KbTreeFolder | KbTreeFile]) -> str | None:
"""A node's ``updated_at`` (phase 106, D9): the subtree's MAX
document ``created_at``, computed from the node's direct children —
files contribute their ``created_at``, subfolder nodes contribute
their (already recursive) ``updated_at``.
All values share the same UTC ``isoformat()`` shape (the endpoint
converts every ``created_at`` before the builder runs), so the
LEXICOGRAPHIC max is the chronological max — ISO-8601 strings of
one offset order by instant. ``None`` when no child carries a date
(a node with no documents at all — the 0-document source).
"""
dates = [
child.created_at if child.kind == "file" else child.updated_at
for child in children
]
dates = [d for d in dates if d is not None]
return max(dates) if dates else None
def build_kb_tree(
names: Sequence[str],
doc_rows: Sequence[TreeDocRow],
@@ -413,8 +537,9 @@ def build_kb_tree(
*names* — the registry source names in order (``app.rag.agent.
list_source_names`` — deduped, registry order). *doc_rows* — the
catalogue ``(source, path, title, chunks, indexed_at)`` tuples in
the ``GET /api/docs`` query order (``source, path``). *summaries* —
catalogue ``(source, path, title, chunks, indexed_at, created_at)``
tuples (both stamps ISO-8601) in the ``GET /api/docs`` query order
(``source, path``). *summaries* —
``{(source, folder_path): summary}`` over the stored
``folder_summaries`` rows (``folder_path = ""`` = the source root;
rows for sources the tree does not list are simply never
@@ -433,13 +558,21 @@ def build_kb_tree(
source node IS the root); direct subfolders only, in path
(sorted) order, a folder existing only under the phase-94
existence rule; ``documents`` = the recursive subtree count;
``summary`` = the stored row (AI OR manual — any row) or null;
``children`` = the folder's own subfolders + direct files, same
shape.
``updated_at`` (phase 106, D9) = the subtree's MAX document
``created_at`` — DERIVED as the builder recurses (the max of the
direct files' dates and the children's ``updated_at`` values via
:func:`_subtree_max`; the ISO-8601 strings share one
``isoformat()`` shape, so the lexicographic max is the
chronological one), never stored — ``null`` when the node has no
documents at all; ``summary`` = the stored row (AI OR manual —
any row) or null; ``children`` = the folder's own subfolders +
direct files, same shape.
* **File nodes** — direct files only, in input (catalog) order;
``path`` source-relative; ``title`` / ``chunks`` / ``indexed_at``
verbatim from the catalogue row. File nodes carry NO pending
flag (the file table has no description column).
``path`` source-relative; ``title`` / ``chunks`` / ``created_at``
(phase 106) / ``indexed_at`` verbatim from the catalogue row.
File nodes carry NO pending flag (the file table has no
description column) and no ``updated_at`` (a file's date IS its
``created_at``).
* **Pending** — ``summary_pending`` on the SOURCE and every FOLDER
node (phase 98, decision D3 — ONE concept): true iff the node's
recursive ``documents`` count ≥
@@ -460,13 +593,20 @@ def build_kb_tree(
existence / count rules, so — for a single-source dataset — its
level equals :func:`app.rag.agent.group_folder_listing`'s output
(same subfolder ``(path, count, summary)`` triples in order, same
file ``(path, title)`` pairs in order — the "UI shows what the
agent sees" cross-check, unit-pinned at the root and a nested
level).
file ``(source, path, title, date)`` 4-tuples in order — phase
106, D5: the agent's file lines carry the appended ``date`` field
and the tree's file nodes carry ``created_at``; the "UI shows
what the agent sees" cross-check, unit-pinned at the root and a
nested level, compares the extended shapes).
D9 (phase 106): ``updated_at`` on every SOURCE and FOLDER node is
the subtree's MAX document ``created_at`` — derived, never stored;
``None`` for a node with no documents at all (the registered
0-document source).
"""
by_source: dict[str, list[TreeFileRow]] = {}
for source, path, title, chunks, indexed_at in doc_rows:
by_source.setdefault(source, []).append((path, title, chunks, indexed_at))
for source, path, title, chunks, indexed_at, created_at in doc_rows:
by_source.setdefault(source, []).append((path, title, chunks, indexed_at, created_at))
tree: list[KbTreeSource] = []
listed: set[str] = set()
for name in names:
@@ -499,14 +639,20 @@ def _source_node(
``summary_pending`` (phase 98, D3): the whole-source count ≥
:data:`MIN_DOCS_PER_FOLDER` AND no stored ``(source, "")`` row —
the source-root arm of the rule :func:`build_kb_tree` documents.
``updated_at`` (phase 106, D9): the source's subtree max —
:func:`_subtree_max` over the root level's children; ``None`` for a
0-document source (no children, no dates).
"""
folders, counts = _folder_counts(rows)
children = _level_children(source, "", folders, counts, rows, summaries)
return KbTreeSource(
name=source,
documents=len(rows),
updated_at=_subtree_max(children),
summary=summaries.get((source, "")),
summary_pending=len(rows) >= MIN_DOCS_PER_FOLDER and (source, "") not in summaries,
children=_level_children(source, "", folders, counts, rows, summaries),
children=children,
)
@@ -540,15 +686,23 @@ def list_kb_tree(
Document.path,
Document.title,
func.count(Chunk.id).label("chunks"),
Document.created_at,
Document.indexed_at,
)
.outerjoin(Chunk, Chunk.document_id == Document.id)
.group_by(Document.id, Document.source, Document.path, Document.title, Document.indexed_at)
.group_by(
Document.id,
Document.source,
Document.path,
Document.title,
Document.created_at,
Document.indexed_at,
)
.order_by(Document.source, Document.path)
).all()
doc_rows: list[TreeDocRow] = [
(source, path, title, chunks, indexed_at.isoformat())
for source, path, title, chunks, indexed_at in rows
(source, path, title, chunks, indexed_at.isoformat(), created_at.isoformat())
for source, path, title, chunks, created_at, indexed_at in rows
]
summaries: dict[tuple[str, str], str] = {
(source, folder_path): summary
+18 -4
View File
@@ -36,9 +36,13 @@ decisions):
4. ``import_sources(..., prune=True)`` over the single combined list
(git checkouts + local dirs), honoring each row's ``ignore_paths``
(phase 89 — the per-root ignore map is built in the same per-row
loop as the source list) and its ``include_hidden`` flag (phase 105
— the per-root hidden-folders map, same per-row construction) —
prune so files deleted upstream, out of
loop as the source list), its ``include_hidden`` flag (phase 105 —
the per-root hidden-folders map, same per-row construction), and
feeding each git checkout's per-file last-commit dates (phase 106,
D2 — the per-root date map, ``file_commit_dates`` after the clone,
same per-row construction; local rows contribute nothing and take
the importer's mtime fallback) — prune so files deleted upstream,
out of
a local dir, or newly matching an ignore pattern leave the index
(pruning covers the union; the CLI's no-prune default is unchanged);
5. when the import changed the KB (added + updated > 0),
@@ -117,7 +121,7 @@ from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient, check_models
from app.rag.overview import regenerate_overview
from app.rag.sources_meta import bump_sources_version, current_sources_version
from scripts.git_sync import GitSyncError, clone_or_pull
from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates
from scripts.import_docs import repo_name
logger = logging.getLogger("app.api.sync")
@@ -286,9 +290,17 @@ async def _run_sync() -> None:
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
include_hidden_by_root: dict[str, bool] = {}
doc_dates_by_root: dict[str, dict[str, datetime]] = {}
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
# Phase 106 (D2): the checkout's per-file last-commit
# dates, keyed by the SAME root string the importer
# sees (local-path checkouts → true per-file dates,
# shallow URL checkouts → the tip date, D10). Local
# rows contribute nothing — the importer's mtime
# fallback applies to them.
doc_dates_by_root[str(root)] = file_commit_dates(root)
else:
# kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the
@@ -329,6 +341,7 @@ async def _run_sync() -> None:
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
)
overview = False
if summary.added + summary.updated > 0:
@@ -431,6 +444,7 @@ async def _run_sync() -> None:
"chunks": summary.chunks,
"summaries": summary.summaries,
"summary_errors": summary.summary_errors,
"dates_updated": summary.dates_updated,
"overview": overview,
"sources_version": sources_version,
}
+41
View File
@@ -185,6 +185,28 @@ class Settings(BaseSettings):
hybrid_vector_candidates: int = 100
hybrid_lexical_candidates: int = 30
rrf_k: int = 60
#: Recency boost on the RRF-fused retrieval score (phase 106, D6): the
#: MAXIMUM additive score a zero-age document gets —
#: ``fused + recency_boost * exp(-age_days / recency_half_life_days)``
#: (``app.rag.retriever.apply_recency_boost``, applied in
#: ``retrieve()`` after ``fuse()``). ``0`` = off — the pre-phase
#: ranking is byte-identical (the kill switch); negative values fail
#: startup loudly (the ``agent_max_rounds`` validator pattern).
#: 0.0007 ≈ a 2-3 rank head start on a 60+ RRF scale (rank 1 vs 2
#: in one list differs by ~0.00026, rank 1 vs 10 by ~0.0021) —
#: enough to break near-ties toward the newer document, far below
#: the gap between a document that answers and one that merely
#: resembles (the phase-106 fine-line battery pins the measured
#: margin). The design starting point was 0.001; the battery's
#: 3×-margin requirement tuned it down here (task 07 step 5) — on
#: the k=60 scale a 0.001 boost would flip the pinned owner
#: scenario (older-correct vs newer-similar).
recency_boost: float = 0.0007
#: Age (days) over which the recency boost decays (phase 106, D6):
#: the boost multiplies by ``e**-1`` ≈ 0.37 per ``recency_half_life_days``
#: of document age (full weight at age 0, ``weight/e`` at one
#: half-life). ``<= 0`` fails startup loudly (same validator family).
recency_half_life_days: int = 365
# --- Admin & sign-in (phase 16; A10 revised 2026-08-22) ---
# Single-admin auth via a signed session cookie (Starlette
@@ -351,6 +373,25 @@ class Settings(BaseSettings):
raise ValueError("history_max_chars must be >= 0 (chars)")
return v
@field_validator("recency_boost")
@classmethod
def _recency_boost_non_negative(cls, v: float) -> float:
"""``0`` is the kill switch (pre-phase ranking byte-identical) — a
negative boost would demote fresh documents, the exact opposite
of D6 (the ``agent_max_rounds`` pattern, phase 106)."""
if v < 0:
raise ValueError("recency_boost must be >= 0 (0 = off)")
return v
@field_validator("recency_half_life_days")
@classmethod
def _recency_half_life_days_positive(cls, v: int) -> int:
"""``0``/negative would divide the decay exponent by zero — fail
loud at startup (the ``agent_max_rounds`` pattern, phase 106)."""
if v <= 0:
raise ValueError("recency_half_life_days must be > 0 (days)")
return v
@field_validator("docs_branch", "docs_base_branch")
@classmethod
def _docs_branch_tokens(cls, v: str, info: ValidationInfo) -> str:
+21
View File
@@ -110,6 +110,27 @@ class Document(Base):
content: Mapped[str] = mapped_column(Text) # full markdown — the RAG context
content_hash: Mapped[str] = mapped_column(String(64), index=True) # sha256 for change detection
indexed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
#: The document's CREATION date (phase 106, D1/D2/D3) — sourced at
#: sync time (git last-commit date for git sources, file mtime for
#: local dirs / unpacked uploads), normalized by
#: :func:`app.rag.doc_dates.normalize_doc_date` (undetermined or
#: future → today; UTC). NOT NULL: pre-phase-106 rows backfill to
#: the migration moment (≈ today — the owner's instruction) and the
#: next sync refreshes them (the importer's unchanged path,
#: task 04 — a sync may move a date OLDER, D4). Distinct from
#: ``indexed_at`` (the INDEX time, untouched).
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
#: True only while ``created_at`` is the OWNER'S correction (phase
#: 106, D1 — the ``folder_summaries.manually_edited`` phase-97
#: precedent): set ONLY by ``PATCH /api/documents/date``
#: (task 05); the sync-time importer SKIPS the refresh on a manual
#: row (the correction survives syncs, D4) and a content change
#: RESETS both the date and the flag (a new version = a new date).
created_at_manual: Mapped[bool] = mapped_column(
Boolean, default=False, server_default=text("false"), nullable=False
)
#: Lite-model summary, phase 30. Natural-language summary of the
#: document (non-markdown A9 docs only, generated at import time by the
#: aipi ``lite`` model). NULL for markdown docs, pre-phase-30 rows, and
+60 -36
View File
@@ -70,9 +70,11 @@ task 04):
(the count is the subfolder's recursive subtree — every document
whose path equals the folder or starts with ``folder + "/"``, the
same set the sync-time folder summary describes — and the file
lines ``source: X | path: Y | title: Z`` (the canonical
``read``/``grep`` identity — the phase-63 labeled format,
unchanged) in path order (``GET /api/docs`` order), capped at
lines ``source: X | path: Y | title: Z | date: YYYY-MM-DD`` (the
canonical ``read``/``grep`` identity — the phase-63 labeled format —
plus the phase-106 D5 ``date`` field APPENDED after ``title``; only
FILE lines carry a date — source/folder lines are not documents)
in path order (``GET /api/docs`` order), capped at
:data:`LS_MAX_FILE_LINES` lines + one deterministic grep-pointer
note for the rest (a 500-file folder costs 50 lines, never 500);
a ``source/folder`` ``path``: that folder's subfolders + own file
@@ -463,7 +465,8 @@ NO_DOCUMENT_DID_YOU_MEAN_MANY = (
SUGGESTION_LIMIT = 3
#: The drill-down ``ls`` file-line cap (phase 94, task 03): a folder's
#: own files list at most this many ``source: X | path: Y | title: Z``
#: own files list at most this many
#: ``source: X | path: Y | title: Z | date: YYYY-MM-DD``
#: lines (path order), then one deterministic grep-pointer note — a
#: 500-file folder costs the model 50 lines + the note, never 500.
#: Pinned module constant (no env var — the phase-94 TODO asks for a
@@ -655,14 +658,17 @@ def _source_root_summaries(db: Session) -> list[tuple[str, str]]:
]
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str]]:
"""``(path, title)`` of every document under *source*, ordered by
``path`` — the one bounded fetch a folder drill level lists (phase
94 task 03; one source's paths, not the whole KB)."""
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str, str]]:
"""``(path, title, created_iso_date)`` of every document under
*source*, ordered by ``path`` — the one bounded fetch a folder
drill level lists (phase 94 task 03; one source's paths, not the
whole KB). The date is the row's ``created_at`` UTC date part
(``YYYY-MM-DD``, phase 106 D5 — the ``ls`` FILE line's appended
`` | date: …`` field; only file lines carry a date)."""
return [
(path, title)
for path, title in db.execute(
select(Document.path, Document.title)
(path, title, created_at.strftime("%Y-%m-%d"))
for path, title, created_at in db.execute(
select(Document.path, Document.title, Document.created_at)
.where(Document.source == source)
.order_by(Document.path)
)
@@ -707,15 +713,17 @@ def ls_top(db: Session) -> list[tuple[str, int, str | None]]:
def group_folder_listing(
source: str,
folder: str,
rows: Sequence[tuple[str, str]],
rows: Sequence[tuple[str, str, str]],
summaries: Mapping[str, str],
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
) -> tuple[
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
]:
"""One level of the drill-down tree (phase 94, task 03) — pure.
Given *rows* — the source's ``(path, title)`` pairs in catalog
(path) order — and *summaries* (the source's stored
``folder_summaries`` rows: ``folder_path → summary``), the folder
level *folder* (source-relative; ``""`` = the source root):
Given *rows* — the source's ``(path, title, created_iso_date)``
triples in catalog (path) order — and *summaries* (the source's
stored ``folder_summaries`` rows: ``folder_path → summary``), the
folder level *folder* (source-relative; ``""`` = the source root):
* **(a) direct subfolders** — the folders whose parent is exactly
*folder*, in path order, each
@@ -732,8 +740,10 @@ def group_folder_listing(
prefix before the last ``/`` —
:func:`app.rag.folder_summaries.folder_of`, the shared notion) IS
*folder*, in path order (catalog order — the same order
``GET /api/docs`` serves), as ``(source, path, title)`` triples
— the canonical ``read``/``grep`` identity, capped at
``GET /api/docs`` serves), as ``(source, path, title, date)``
4-tuples — the canonical ``read``/``grep`` identity plus the
phase-106 D5 ``date`` field (the row's ``created_at`` UTC date
part, APPENDED — never inserted before ``title``), capped at
:data:`LS_MAX_FILE_LINES` (the rest fold into the renderer's
note; a 500-file folder never costs 500 lines).
* **(c) the TOTAL direct-file count** — pre-cap, for the note.
@@ -745,7 +755,7 @@ def group_folder_listing(
# indexed path (the existence rule's candidate set — a folder is
# present iff at least one path starts with ``folder + "/"``).
folders: set[str] = set()
for path, _title in rows:
for path, _title, _date in rows:
f = folder_of(path)
while f:
folders.add(f)
@@ -755,7 +765,7 @@ def group_folder_listing(
# folder + "/"`` arm (the folder's true descendants), one pass per
# document.
counts: dict[str, int] = {f: 0 for f in folders}
for path, _title in rows:
for path, _title, _date in rows:
if path in folders:
counts[path] += 1
f = folder_of(path)
@@ -767,8 +777,8 @@ def group_folder_listing(
for g in sorted(g for g in folders if folder_of(g) == folder)
]
files = [
(source, path, title)
for path, title in rows
(source, path, title, date)
for path, title, date in rows
if folder_of(path) == folder
]
return subfolders, files[:LS_MAX_FILE_LINES], len(files)
@@ -776,7 +786,9 @@ def group_folder_listing(
def ls_folder(
db: Session, source: str, folder: str
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
) -> tuple[
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
]:
"""One folder level of the drill-down ``ls`` (phase 94, task 03).
The source's document rows (:func:`_source_document_rows`) and
@@ -793,7 +805,7 @@ def ls_folder(
)
def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
def _folder_exists_in(rows: Sequence[tuple[str, str, str]], folder: str) -> bool:
"""The phase-94 folder-existence rule (``00_phase.md``), pure.
Folder *folder* (source-relative) under a registered source
@@ -805,11 +817,11 @@ def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
if not folder:
return True
prefix = folder + "/"
return any(path.startswith(prefix) for path, _title in rows)
return any(path.startswith(prefix) for path, _title, _date in rows)
def _deepest_existing_ancestor(
rows: Sequence[tuple[str, str]], folder: str
rows: Sequence[tuple[str, str, str]], folder: str
) -> str:
"""The deepest EXISTING folder prefix of a missing *folder* (pure).
@@ -854,7 +866,7 @@ def render_ls_top(entries: Sequence[tuple[str, int, str | None]]) -> str:
def render_folder_listing(
identity: str,
subfolders: Sequence[tuple[str, int, str | None]],
files: Sequence[tuple[str, str, str]],
files: Sequence[tuple[str, str, str, str]],
total_files: int,
) -> str:
"""One folder level of the drill-down ``ls`` (phase 94, task 03) —
@@ -867,9 +879,10 @@ def render_folder_listing(
below the header — a blank line, the 2-space-indented subfolder
lines `` {sub}/ — {m} documents`` in path order (``: {summary}``
appended ONLY when the subfolder's summary is stored), a blank
line, the file lines in EXACTLY the existing
``source: X | path: Y | title: Z`` format (the canonical
``read``/``grep`` identity — unchanged), and the cap note
line, the file lines in EXACTLY the
``source: X | path: Y | title: Z | date: YYYY-MM-DD`` format (the
canonical ``read``/``grep`` identity plus the phase-106 D5
appended ``date`` field — the only changed part), and the cap note
``…and {hidden} more documents in this folder — use grep
(pattern) to find a specific one.`` ONLY when the folder's own
files outnumber :data:`LS_MAX_FILE_LINES` (*files* arrives capped;
@@ -891,8 +904,8 @@ def render_folder_listing(
if files or total_files > len(files):
body.append("")
body.extend(
f"source: {source} | path: {path} | title: {title}"
for source, path, title in files
f"source: {source} | path: {path} | title: {title} | date: {date}"
for source, path, title, date in files
)
hidden = total_files - len(files)
if hidden > 0:
@@ -1172,15 +1185,26 @@ def _execute_tool(
holder.read_truncations.append(
(cast("str", raw_path), cap, len(doc.content))
)
# Phase 106 (D5): the date rides every document the model
# sees — the ``read`` result's SECOND line; the FIRST line
# stays ``Document {source}/{path}:`` BYTE-IDENTICAL (the
# E2E mock's ``_READ_RESULT_PREFIX`` header contract).
return (
f"Document {doc.source}/{doc.path}:\n"
f"date: {doc.created_at:%Y-%m-%d}\n"
f"{doc.content[:cap]}\n"
f"{TRUNCATION_MARKER}\n"
f"{READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))}"
)
# At or under the cap: byte-identical to the pre-phase-95 result
# (no marker, no notice, no holder entry, no ToolResultPiece).
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
# At or under the cap: the pre-phase-95 result plus the
# phase-106 D5 date line (first line byte-identical — the
# mock's header contract; no marker, no notice, no holder
# entry, no ToolResultPiece).
return (
f"Document {doc.source}/{doc.path}:\n"
f"date: {doc.created_at:%Y-%m-%d}\n"
f"{doc.content}"
)
if call.name == "grep":
raw_pattern = call.arguments.get("pattern")
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
+26
View File
@@ -18,6 +18,13 @@ The guarantees (phase 49 locked decisions):
unpack directory, and device/FIFO members — and counting every
extracted byte against a cap (zip-bomb guard). Any failure removes the
partial ``target_dir`` so no half-unpacked tree survives.
* :func:`unpack_archive` also restores each regular file's member mtime
(the zip DOS ``date_time`` or the tar ``mtime``) — phase 106, D2:
uploaded archives keep their file dates, which the importer reads as
the document creation date. Directories, symlinks, and hardlinks are
untouched; the cap and every safety check above are unchanged (the
``utime`` sits after a successful ``_write_capped``, so a failed
unpack still removes the partial tree).
* :func:`swap_in` makes ``new_dir`` become ``final_dir`` with **no
missing window**: the previous folder is renamed to a unique
same-filesystem ``.old-`` sibling first, the new folder is renamed
@@ -33,6 +40,7 @@ import stat
import tarfile
import uuid
import zipfile
from datetime import UTC, datetime
from pathlib import Path, PurePosixPath
from typing import IO
@@ -183,6 +191,16 @@ def _unpack_zip(archive: Path, target: Path, max_extract_bytes: int) -> None:
else:
with zf.open(member) as src:
_write_capped(src, dest, max_extract_bytes, total)
# Phase 106 (D2): restore the member's DOS mtime — a
# tz-agnostic epoch value, UTC-rendered exactly like an
# mtime — so uploaded archives keep their file dates.
# Regular files only; an OSError here still removes the
# partial tree in unpack_archive like any write failure.
# ``ns=`` takes INTEGER nanoseconds (a float seconds
# value raises TypeError), so convert explicitly.
mtime = datetime(*member.date_time, tzinfo=UTC).timestamp()
mtime_ns = int(mtime * 1_000_000_000)
os.utime(dest, ns=(mtime_ns, mtime_ns))
def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
@@ -206,6 +224,14 @@ def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
if src is None:
raise ArchiveUploadError("corrupt archive member")
_write_capped(src, dest, max_extract_bytes, total)
# Phase 106 (D2): restore the member's mtime (epoch
# seconds — a tz-agnostic value) so uploaded archives
# keep their file dates. Regular files only; an OSError
# here still removes the partial tree in unpack_archive.
# ``ns=`` takes INTEGER nanoseconds (a float seconds
# value raises TypeError), so convert explicitly.
mtime_ns = int(member.mtime * 1_000_000_000)
os.utime(dest, ns=(mtime_ns, mtime_ns))
else: # char/block device, FIFO
raise ArchiveUploadError(
"tar archives with device or FIFO members are not allowed"
+65
View File
@@ -0,0 +1,65 @@
"""Document-creation-date sourcing + normalization (phase 106, D2/D3).
Every document date the importer writes and every date the owner
edits passes through :func:`normalize_doc_date` — the single choke
point for the owner's rules: an UNDETERMINED date (no source signal)
and a FUTURE date (beyond a small clock-skew tolerance) both assume
the document was created TODAY (UTC). Naive source timestamps (zip
DOS mtimes, tar mtimes, git-free fallbacks) are tz-agnostic epoch-
based values rendered as UTC; aware ones are converted to UTC.
Pure and stdlib-only by contract (unit-pinned in
``tests/unit/test_doc_dates.py``): no database, no logging, no I/O
besides :func:`file_mtime_datetime`'s single ``stat`` — the callers
(importer task 04, the date-edit API task 05) own everything else.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from pathlib import Path
__all__ = ["FUTURE_SKEW_TOLERANCE", "file_mtime_datetime", "normalize_doc_date"]
#: Clock-skew tolerance (D3): a source date up to this far in the
#: FUTURE is a drifting clock, not a future document — it keeps its
#: date. Beyond it, the owner's rule applies (→ today).
FUTURE_SKEW_TOLERANCE = timedelta(days=1)
def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime:
"""*raw* → the stored UTC creation date (the D3 rule, pinned).
``now`` is injectable (tests); it defaults to
``datetime.now(UTC)``. ``raw=None`` (undetermined) → *now*;
naive *raw* → treated as UTC; aware *raw* → converted to UTC;
*raw* beyond *now* + :data:`FUTURE_SKEW_TOLERANCE` → *now*.
The result always carries full precision (no date-truncation —
the display formats, the storage doesn't).
"""
if now is None:
now = datetime.now(UTC)
elif now.tzinfo is None:
# The future check compares in AWARE space — a naive ``now``
# (callers/tests) is a UTC instant, like the naive ``raw``.
now = now.replace(tzinfo=UTC)
if raw is None:
return now
# Epoch-based source values (zip DOS times, tar mtimes) are
# tz-agnostic — attach UTC; never assume the host's local TZ.
# Aware values are converted to UTC (the comparison below is
# done in aware space).
raw = raw.replace(tzinfo=UTC) if raw.tzinfo is None else raw.astimezone(UTC)
if raw > now + FUTURE_SKEW_TOLERANCE:
# Genuinely future (beyond the clock-skew tolerance) → today.
return now
return raw
def file_mtime_datetime(path: Path) -> datetime:
"""The file's mtime as an aware UTC datetime (the D2 fallback).
Epoch mtimes are tz-agnostic — UTC is the correct rendering
(zip DOS timestamps and tar mtimes pass through the same
:func:`normalize_doc_date` after unpacking, task 03).
"""
return datetime.fromtimestamp(path.stat().st_mtime, tz=UTC)
+80 -3
View File
@@ -31,6 +31,17 @@ previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
summary line with per-format counts (PLAN §9).
Document dates (phase 106, D2/D4): every import sources
``documents.created_at`` from the file's source — the per-file git
last-commit date when a ``doc_dates_by_root`` entry names the file,
else the file's mtime — normalized by
:func:`app.rag.doc_dates.normalize_doc_date` (undetermined or future →
today, D3) on every add and update. On the unchanged path the stored
date is REFRESHED from the same source (it may go OLDER — no monotonic
guard) and counted in ``summary.dates_updated`` — unless the row
carries the owner's manual correction (``created_at_manual``, D1), which
the sync never touches.
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
"""
@@ -51,6 +62,7 @@ from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.rag.chunker import chunk_document, extract_title
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
from app.rag.llm import EmbeddingError, LLMError
from app.rag.summarizer import generate_summary
@@ -93,6 +105,12 @@ class ImportSummary:
#: Non-markdown files whose summary generation failed (best-effort —
#: the document is still indexed, without a summary).
summary_errors: int = 0
#: Files whose ``created_at`` was refreshed on the UNCHANGED path —
#: content untouched, date re-sourced (phase 106, D4: the date may
#: go OLDER; a date-only refresh NEVER counts added/updated/pruned,
#: so no ``sources_meta`` bump, no overview/folder-summary
#: regeneration).
dates_updated: int = 0
#: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
formats: dict[str, int] = field(default_factory=dict)
@@ -107,7 +125,7 @@ class ImportSummary:
logger.info(
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"formats=%s",
"dates_updated=%d formats=%s",
self.files,
self.added,
self.updated,
@@ -118,6 +136,7 @@ class ImportSummary:
self.embed_batches,
self.summaries,
self.summary_errors,
self.dates_updated,
self.format_counts(),
)
@@ -248,6 +267,7 @@ async def import_sources(
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring).
@@ -288,6 +308,19 @@ async def import_sources(
flag ON and is walked again with it OFF simply never enters
``seen``, so the next ``prune=True`` run deletes its row
automatically (A2 — the A9/phase-89 precedent).
``doc_dates_by_root`` (phase 106, D2/D4) maps ``str(root)`` — the
root path string exactly as passed in *sources* — to that source's
RAW per-file source dates: source-relative POSIX path → the git
last-commit datetime (task 03's ``file_commit_dates``). ONLY git
roots are listed — unlisted roots (local dirs, unpacked uploads)
take the mtime fallback, and a path missing from its root's map
does too. The map entry beats the file's mtime when present. The
progress pre-walk is untouched (dates change no file count).
``None`` (the default) changes nothing for existing callers: the
mtime fallback applies to every file — which IS the behavior
change, D4: an unchanged file now refreshes its stored date from
its source on every run (the backfill-correction case).
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -325,6 +358,10 @@ async def import_sources(
source_names.add(source)
ignore = _ignore_for_root(root, ignore_by_root)
include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
# Phase 106 (D2): the root's raw source dates (git last-commit
# for git roots, keyed by the same str(root) convention); {}
# for unlisted roots — every file then takes the mtime fallback.
dates_map = (doc_dates_by_root or {}).get(str(root), {})
for path in iter_importable_files(
root,
llm.settings.import_extension_set,
@@ -353,7 +390,7 @@ async def import_sources(
try:
await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm,
summary=summary,
summary=summary, raw_date=dates_map.get(rel),
)
except EmbeddingError as e:
# A pathological file (e.g. content the embedding endpoint
@@ -384,15 +421,45 @@ async def _index_file(
full_path: Path,
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
) -> None:
"""Upsert one file: doc row + chunk rows + embeddings, one transaction."""
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
``raw_date`` (phase 106, D2) is the file's RAW source date — the
git last-commit datetime from the caller's ``doc_dates_by_root``
map, or ``None`` (every non-git case): the file's mtime is read
here, once, and becomes the source date (the D2 fallback).
"""
settings = llm.settings
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
if raw_date is None:
# D2 fallback: no source date in the map → the file's mtime
# (one stat). Read before the unchanged early-return — the
# unchanged path refreshes the stored date from the same source.
raw_date = file_mtime_datetime(full_path)
if doc is not None and doc.content_hash == digest:
summary.unchanged += 1
logger.info("import: unchanged source=%s path=%s", source, rel)
if doc.created_at_manual:
# D1/D4: the owner's correction survives the sync — no
# write at all (the phase-97 ``manually_edited`` precedent).
return
# D4: the date refreshes on every sync, including unchanged
# files, and may go OLDER (no monotonic guard). A date-only
# refresh is still counted ``unchanged`` — never added/updated/
# pruned, so no ``sources_meta`` bump and no regeneration.
target = normalize_doc_date(raw_date)
if target != doc.created_at:
doc.created_at = target
session.commit()
summary.dates_updated += 1
logger.info(
"import: date-refreshed source=%s path=%s date=%s",
source, rel,
doc.created_at.isoformat(),
)
return
verb = "updated" if doc is not None else "added"
@@ -411,6 +478,11 @@ async def _index_file(
content=content,
content_hash=digest,
indexed_at=datetime.now(UTC),
# Phase 106 (D2/D3): the sourced creation date, normalized
# (undetermined or future → today). ``created_at_manual``
# stays the column default (False) — only the date-edit API
# (task 05) sets it.
created_at=normalize_doc_date(raw_date),
)
session.add(doc)
else:
@@ -419,6 +491,11 @@ async def _index_file(
doc.content = content
doc.content_hash = digest
doc.indexed_at = datetime.now(UTC)
# Phase 106 (D4): a content change is a new document version —
# the date is re-sourced and a previous manual correction is
# reset (it referred to the old content).
doc.created_at = normalize_doc_date(raw_date)
doc.created_at_manual = False
session.flush() # guarantees doc.id even for brand-new rows
+12 -1
View File
@@ -352,6 +352,12 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
cap — not the prompt — decides whether the tools are actually
offered to the model, see :mod:`app.rag.agent`).
Each ``<document>`` block carries the identity attributes
``source`` / ``path`` / ``title`` — and, since phase 106 (D5),
``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``,
APPENDED after ``title`` — the only position; always present,
``created_at`` is NOT NULL) — plus the document's full text.
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
LEADING this section (the document texts are already context — do
not ``read`` one the user asked to open) was tried and REVERTED:
@@ -368,8 +374,13 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
# attribute at the ``source``/``path`` copy site was TRIED and
# REVERTED the same day (no improvement across runs; the block stays
# exactly the document identity + full text).
# Phase 106 (D5): every document the model sees carries its
# creation date — the block's ``date`` attribute (the row's
# ``created_at`` UTC date part, appended after ``title`` — the
# only position; always present, ``created_at`` is NOT NULL).
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}">\n'
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
f'date="{doc.created_at:%Y-%m-%d}">\n'
f"{doc.content}\n"
"</document>"
for doc in documents
+81 -1
View File
@@ -41,10 +41,12 @@ Deterministic tie-break for equal fused scores:
"""
from __future__ import annotations
import math
import re
import uuid
from collections.abc import Sequence
from dataclasses import dataclass, replace
from datetime import UTC, datetime
from pathlib import Path
from sqlalchemy import select, text
@@ -146,6 +148,7 @@ _LEXICAL_SQL = text(
d.content AS doc_content,
d.content_hash AS content_hash,
d.indexed_at AS indexed_at,
d.created_at AS created_at,
c.is_summary AS is_summary,
ts_rank(c.tsv, to_tsquery('english', :tsquery)) AS rank
FROM chunks c
@@ -242,6 +245,73 @@ def fuse(
return out
def apply_recency_boost(
chunks: Sequence[RetrievedChunk],
*,
now: datetime | None = None,
weight: float | None = None,
half_life_days: int | None = None,
) -> list[RetrievedChunk]:
"""Additive recency boost on the fused score (phase 106, D6).
Each chunk's score becomes
``score + weight * exp(−age_days / half_life_days)`` where
``age_days = max(0, (now − document.created_at))`` in days — a
zero-age document gets the full *weight* (the MAXIMUM additive
score), each ``half_life_days`` of age multiplies the remaining
boost by ``e**-1`` (≈0.37), and a FUTURE ``created_at`` clamps to
age 0 (the document reads as brand-new — consistent with D3's
today-folding in :mod:`app.rag.doc_dates`). Defaults: *weight* /
*half_life_days* from :func:`get_settings` (``recency_boost`` /
``recency_half_life_days``), *now* from ``datetime.now(UTC)``.
Magnitude rationale (the ``0.0007`` default, the k=60 RRF scale):
rank 1 vs 2 in one list differs by ~0.00026 and rank 1 vs 10 by
~0.0021, so the full weight is a bounded 2-3 rank head start —
enough to break near-ties toward the newer document, far below the
fused gap between a document that answers and one that merely
resembles (the phase-106 fine-line battery pins the measured
margin: 0.00263 ≥ 3× the zero-age boost).
Pure (the :func:`fuse` convention): the inputs are never mutated —
every boosted chunk is a ``replace()`` copy — and the result is
re-sorted with the EXISTING deterministic key
``(−score, −cosine, document.path, position)``; with ``weight=0``
every score is untouched and an already-fused (already-sorted)
input comes back byte-identical (the kill switch, pinned).
Untouched by design: the A8 honesty gate and ``query_log.top_score``
(both read the chunk's ``cosine``, which the boost never modifies),
:func:`weak_hit_titles` (titles only), and the never-truncated
top-N contract (:func:`select_documents` still feeds whole
documents — the boost re-ranks WHICH documents, never truncates).
SINGLE APPLY SITE: :func:`retrieve()` is the only caller in
``app/`` — the chat API and ``scripts/eval_retrieval.py`` inherit
the boost through it; nothing else may apply it.
"""
if weight is None or half_life_days is None:
settings = get_settings()
if weight is None:
weight = settings.recency_boost
if half_life_days is None:
half_life_days = settings.recency_half_life_days
if half_life_days <= 0:
raise ValueError("half_life_days must be > 0")
if now is None:
now = datetime.now(UTC)
boosted: list[RetrievedChunk] = []
for rc in chunks:
age_days = max(0.0, (now - rc.document.created_at).total_seconds() / 86400.0)
boosted.append(
replace(
rc,
score=rc.score + weight * math.exp(-age_days / half_life_days),
)
)
boosted.sort(key=lambda rc: (-rc.score, -rc.cosine, rc.document.path, rc.position))
return boosted
def _vector_candidates(
db: Session, question_embedding: list[float], limit: int
) -> list[RetrievedChunk]:
@@ -289,6 +359,7 @@ _NAME_HIT_SQL = text(
d.content AS doc_content,
d.content_hash AS content_hash,
d.indexed_at AS indexed_at,
d.created_at AS created_at,
c.id AS chunk_id,
c.position AS position,
c.content AS content,
@@ -364,6 +435,7 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]:
content=row.doc_content,
content_hash=row.content_hash,
indexed_at=row.indexed_at,
created_at=row.created_at,
)
out.append(
RetrievedChunk(
@@ -418,6 +490,7 @@ def _lexical_candidates(db: Session, question: str, limit: int) -> list[Retrieve
content=row.doc_content,
content_hash=row.content_hash,
indexed_at=row.indexed_at,
created_at=row.created_at,
)
out.append(
RetrievedChunk(
@@ -460,7 +533,14 @@ def retrieve(
raise ValueError("lexical_candidates must be >= 1")
vector = _vector_candidates(db, question_embedding, v_n)
lexical = _lexical_candidates(db, question, l_n)
return fuse(vector, lexical, settings.rrf_k)
fused = fuse(vector, lexical, settings.rrf_k)
if settings.recency_boost > 0:
# Phase 106, D6 — the SINGLE recency-boost apply site: an
# additive post-fusion re-rank (see :func:`apply_recency_boost`).
# ``0`` = off: the pre-phase ranking returns byte-identical (the
# kill switch) and weight-0 callers pay nothing.
return apply_recency_boost(fused)
return fused
def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]:
+72 -7
View File
@@ -235,6 +235,10 @@ class DocSummary(BaseModel):
path: str
title: str
chunks: int
#: The document's creation date (phase 106, D8) — ISO-8601, verbatim
#: from the row (the ``indexed_at`` style); the RAG view's file table
#: renders it as the ``Created`` column (before ``Indexed``).
created_at: str
indexed_at: str
@@ -251,7 +255,8 @@ class KbTreeFile(BaseModel):
JSON shape is the contract). ``path`` is SOURCE-RELATIVE (the RAG
view prefixes the source in its breadcrumb); ``title`` /
``chunks`` (content + ``is_summary`` chunks — the same count
``GET /api/docs`` returns) / ``indexed_at`` (ISO-8601) are verbatim
``GET /api/docs`` returns) / ``created_at`` (phase 106, D8 — the
document's creation date) / ``indexed_at`` (ISO-8601) are verbatim
from the catalogue row the endpoint reads.
"""
@@ -259,6 +264,10 @@ class KbTreeFile(BaseModel):
path: str
title: str
chunks: int = Field(ge=0)
#: The document's creation date (phase 106, D8) — ISO-8601, verbatim
#: from the catalogue row; the RAG view's file table renders it as
#: the ``Created`` column (before ``Indexed``).
created_at: str
indexed_at: str
@@ -269,11 +278,14 @@ class KbTreeFolder(BaseModel):
node IS the root); ``documents`` is the recursive subtree count
(the phase-94 ``ls`` count rule: every path equal to the folder or
starting with ``folder + "/"`` — the file sharing a folder's name
counts); ``summary`` is the stored ``folder_summaries`` row (AI or
manual — any row) or null; ``children`` are the direct subfolders
(path order) followed by the direct files (catalog order) — the
recursive union (Pydantic v2 resolves it with
``from __future__ import annotations``).
counts); ``updated_at`` (phase 106, D9) is the subtree's MAX
document ``created_at`` — DERIVED in the pure tree builder as it
recurses, never stored (``null`` for a node with no documents at
all — the ``summary: str | None`` shape); ``summary`` is the stored
``folder_summaries`` row (AI or manual — any row) or null;
``children`` are the direct subfolders (path order) followed by the
direct files (catalog order) — the recursive union (Pydantic v2
resolves it with ``from __future__ import annotations``).
``summary_pending`` (phase 98, D3 — ONE concept): true iff this
folder's recursive count ≥ ``MIN_DOCS_PER_FOLDER`` (2) AND it has
@@ -287,6 +299,10 @@ class KbTreeFolder(BaseModel):
kind: Literal["folder"] = "folder"
path: str
documents: int = Field(ge=0)
#: The subtree's MAX document ``created_at`` (phase 106, D9 —
#: derived in the pure builder, never stored); ISO-8601, ``null``
#: for a node with no documents at all.
updated_at: str | None = None
summary: str | None = None
summary_pending: bool = False
children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
@@ -299,7 +315,10 @@ class KbTreeSource(BaseModel):
always present, a registered 0-document source lists with
``documents: 0`` and no children), then the indexed-only sources
(alphabetical) — the phase-97 superset rule. ``documents`` is the
source's whole recursive count; ``summary`` is the stored
source's whole recursive count; ``updated_at`` (phase 106, D9) is
the source's subtree MAX document ``created_at`` — DERIVED in the
pure tree builder, never stored (``null`` for a 0-document source —
the ``summary: str | None`` shape); ``summary`` is the stored
``(source, "")`` source-root row or null; ``children`` are the
source's direct subfolders + direct files (same shape as a folder
node's).
@@ -316,6 +335,10 @@ class KbTreeSource(BaseModel):
name: str
documents: int = Field(ge=0)
#: The source's subtree MAX document ``created_at`` (phase 106,
#: D9 — derived in the pure builder, never stored); ISO-8601,
#: ``null`` for a 0-document source.
updated_at: str | None = None
summary: str | None = None
summary_pending: bool = False
children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
@@ -350,6 +373,10 @@ class DocContent(BaseModel):
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
#: summary generation failed but the document was still indexed.
summary: str | None = None
#: The document's creation date (phase 106, D8) — ISO-8601, verbatim
#: from the row; the viewer's top meta row renders the ``Created``
#: badge from it (before the ``Indexed`` badge).
created_at: str
content: str
indexed_at: str
chunks: int
@@ -389,6 +416,44 @@ class SummaryResult(BaseModel):
chunks: int
class DateUpdate(BaseModel):
"""``PATCH /api/documents/date`` body (phase 106, task 05, D7).
``source`` / ``path`` name the indexed document (the same pair the
public ``GET /api/documents/content`` looks up); ``date`` is the
owner's corrected creation date — an ISO date (``YYYY-MM-DD``) or a
full ISO datetime. **Null/absent is the CLEAR** (the "revert to
sync" operation, D7): the ``created_at_manual`` flag is dropped and
the stored date stands until the next sync refreshes it (the API is
DB-only — it cannot re-read the source). Unconstrained
``str | None`` on purpose: a MALFORMED non-null value 422s in the
handler (``datetime.fromisoformat``), so the error detail can name
the field; an unknown pair must 404 as "document not found"
(row-lookup semantics), exactly like the public content endpoint.
"""
source: str
path: str
date: str | None = None
class DateResult(BaseModel):
"""``PATCH /api/documents/date`` response (phase 106, task 05, D7).
Echoes the STORED state after the change: ``created_at`` is the
stored ISO-8601 value (a set stores the normalized parse — a
manually set FUTURE date folds to today, D3; a clear leaves the
stored date standing) and ``created_at_manual`` the flag (true
after a set, false after a clear). The viewer re-renders its
Created badge from exactly this echo — no second fetch.
"""
source: str
path: str
created_at: str
created_at_manual: bool
class FolderSummaryUpdate(BaseModel):
"""``PATCH /api/folders/summary`` body (phase 97, task 03).