phase: 106_document_dates
Everything is verified green. Final report: **Phase 106 — Document dates (final verification pass; all 10 tasks already complete)** - Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed. - `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓) - `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up) - 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation** - `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings** **Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions. - **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring). - **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+60
-36
@@ -70,9 +70,11 @@ task 04):
|
||||
(the count is the subfolder's recursive subtree — every document
|
||||
whose path equals the folder or starts with ``folder + "/"``, the
|
||||
same set the sync-time folder summary describes — and the file
|
||||
lines ``source: X | path: Y | title: Z`` (the canonical
|
||||
``read``/``grep`` identity — the phase-63 labeled format,
|
||||
unchanged) in path order (``GET /api/docs`` order), capped at
|
||||
lines ``source: X | path: Y | title: Z | date: YYYY-MM-DD`` (the
|
||||
canonical ``read``/``grep`` identity — the phase-63 labeled format —
|
||||
plus the phase-106 D5 ``date`` field APPENDED after ``title``; only
|
||||
FILE lines carry a date — source/folder lines are not documents)
|
||||
in path order (``GET /api/docs`` order), capped at
|
||||
:data:`LS_MAX_FILE_LINES` lines + one deterministic grep-pointer
|
||||
note for the rest (a 500-file folder costs 50 lines, never 500);
|
||||
a ``source/folder`` ``path``: that folder's subfolders + own file
|
||||
@@ -463,7 +465,8 @@ NO_DOCUMENT_DID_YOU_MEAN_MANY = (
|
||||
SUGGESTION_LIMIT = 3
|
||||
|
||||
#: The drill-down ``ls`` file-line cap (phase 94, task 03): a folder's
|
||||
#: own files list at most this many ``source: X | path: Y | title: Z``
|
||||
#: own files list at most this many
|
||||
#: ``source: X | path: Y | title: Z | date: YYYY-MM-DD``
|
||||
#: lines (path order), then one deterministic grep-pointer note — a
|
||||
#: 500-file folder costs the model 50 lines + the note, never 500.
|
||||
#: Pinned module constant (no env var — the phase-94 TODO asks for a
|
||||
@@ -655,14 +658,17 @@ def _source_root_summaries(db: Session) -> list[tuple[str, str]]:
|
||||
]
|
||||
|
||||
|
||||
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str]]:
|
||||
"""``(path, title)`` of every document under *source*, ordered by
|
||||
``path`` — the one bounded fetch a folder drill level lists (phase
|
||||
94 task 03; one source's paths, not the whole KB)."""
|
||||
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str, str]]:
|
||||
"""``(path, title, created_iso_date)`` of every document under
|
||||
*source*, ordered by ``path`` — the one bounded fetch a folder
|
||||
drill level lists (phase 94 task 03; one source's paths, not the
|
||||
whole KB). The date is the row's ``created_at`` UTC date part
|
||||
(``YYYY-MM-DD``, phase 106 D5 — the ``ls`` FILE line's appended
|
||||
`` | date: …`` field; only file lines carry a date)."""
|
||||
return [
|
||||
(path, title)
|
||||
for path, title in db.execute(
|
||||
select(Document.path, Document.title)
|
||||
(path, title, created_at.strftime("%Y-%m-%d"))
|
||||
for path, title, created_at in db.execute(
|
||||
select(Document.path, Document.title, Document.created_at)
|
||||
.where(Document.source == source)
|
||||
.order_by(Document.path)
|
||||
)
|
||||
@@ -707,15 +713,17 @@ def ls_top(db: Session) -> list[tuple[str, int, str | None]]:
|
||||
def group_folder_listing(
|
||||
source: str,
|
||||
folder: str,
|
||||
rows: Sequence[tuple[str, str]],
|
||||
rows: Sequence[tuple[str, str, str]],
|
||||
summaries: Mapping[str, str],
|
||||
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
|
||||
) -> tuple[
|
||||
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
|
||||
]:
|
||||
"""One level of the drill-down tree (phase 94, task 03) — pure.
|
||||
|
||||
Given *rows* — the source's ``(path, title)`` pairs in catalog
|
||||
(path) order — and *summaries* (the source's stored
|
||||
``folder_summaries`` rows: ``folder_path → summary``), the folder
|
||||
level *folder* (source-relative; ``""`` = the source root):
|
||||
Given *rows* — the source's ``(path, title, created_iso_date)``
|
||||
triples in catalog (path) order — and *summaries* (the source's
|
||||
stored ``folder_summaries`` rows: ``folder_path → summary``), the
|
||||
folder level *folder* (source-relative; ``""`` = the source root):
|
||||
|
||||
* **(a) direct subfolders** — the folders whose parent is exactly
|
||||
*folder*, in path order, each
|
||||
@@ -732,8 +740,10 @@ def group_folder_listing(
|
||||
prefix before the last ``/`` —
|
||||
:func:`app.rag.folder_summaries.folder_of`, the shared notion) IS
|
||||
*folder*, in path order (catalog order — the same order
|
||||
``GET /api/docs`` serves), as ``(source, path, title)`` triples
|
||||
— the canonical ``read``/``grep`` identity, capped at
|
||||
``GET /api/docs`` serves), as ``(source, path, title, date)``
|
||||
4-tuples — the canonical ``read``/``grep`` identity plus the
|
||||
phase-106 D5 ``date`` field (the row's ``created_at`` UTC date
|
||||
part, APPENDED — never inserted before ``title``), capped at
|
||||
:data:`LS_MAX_FILE_LINES` (the rest fold into the renderer's
|
||||
note; a 500-file folder never costs 500 lines).
|
||||
* **(c) the TOTAL direct-file count** — pre-cap, for the note.
|
||||
@@ -745,7 +755,7 @@ def group_folder_listing(
|
||||
# indexed path (the existence rule's candidate set — a folder is
|
||||
# present iff at least one path starts with ``folder + "/"``).
|
||||
folders: set[str] = set()
|
||||
for path, _title in rows:
|
||||
for path, _title, _date in rows:
|
||||
f = folder_of(path)
|
||||
while f:
|
||||
folders.add(f)
|
||||
@@ -755,7 +765,7 @@ def group_folder_listing(
|
||||
# folder + "/"`` arm (the folder's true descendants), one pass per
|
||||
# document.
|
||||
counts: dict[str, int] = {f: 0 for f in folders}
|
||||
for path, _title in rows:
|
||||
for path, _title, _date in rows:
|
||||
if path in folders:
|
||||
counts[path] += 1
|
||||
f = folder_of(path)
|
||||
@@ -767,8 +777,8 @@ def group_folder_listing(
|
||||
for g in sorted(g for g in folders if folder_of(g) == folder)
|
||||
]
|
||||
files = [
|
||||
(source, path, title)
|
||||
for path, title in rows
|
||||
(source, path, title, date)
|
||||
for path, title, date in rows
|
||||
if folder_of(path) == folder
|
||||
]
|
||||
return subfolders, files[:LS_MAX_FILE_LINES], len(files)
|
||||
@@ -776,7 +786,9 @@ def group_folder_listing(
|
||||
|
||||
def ls_folder(
|
||||
db: Session, source: str, folder: str
|
||||
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
|
||||
) -> tuple[
|
||||
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
|
||||
]:
|
||||
"""One folder level of the drill-down ``ls`` (phase 94, task 03).
|
||||
|
||||
The source's document rows (:func:`_source_document_rows`) and
|
||||
@@ -793,7 +805,7 @@ def ls_folder(
|
||||
)
|
||||
|
||||
|
||||
def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
|
||||
def _folder_exists_in(rows: Sequence[tuple[str, str, str]], folder: str) -> bool:
|
||||
"""The phase-94 folder-existence rule (``00_phase.md``), pure.
|
||||
|
||||
Folder *folder* (source-relative) under a registered source
|
||||
@@ -805,11 +817,11 @@ def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
|
||||
if not folder:
|
||||
return True
|
||||
prefix = folder + "/"
|
||||
return any(path.startswith(prefix) for path, _title in rows)
|
||||
return any(path.startswith(prefix) for path, _title, _date in rows)
|
||||
|
||||
|
||||
def _deepest_existing_ancestor(
|
||||
rows: Sequence[tuple[str, str]], folder: str
|
||||
rows: Sequence[tuple[str, str, str]], folder: str
|
||||
) -> str:
|
||||
"""The deepest EXISTING folder prefix of a missing *folder* (pure).
|
||||
|
||||
@@ -854,7 +866,7 @@ def render_ls_top(entries: Sequence[tuple[str, int, str | None]]) -> str:
|
||||
def render_folder_listing(
|
||||
identity: str,
|
||||
subfolders: Sequence[tuple[str, int, str | None]],
|
||||
files: Sequence[tuple[str, str, str]],
|
||||
files: Sequence[tuple[str, str, str, str]],
|
||||
total_files: int,
|
||||
) -> str:
|
||||
"""One folder level of the drill-down ``ls`` (phase 94, task 03) —
|
||||
@@ -867,9 +879,10 @@ def render_folder_listing(
|
||||
below the header — a blank line, the 2-space-indented subfolder
|
||||
lines `` {sub}/ — {m} documents`` in path order (``: {summary}``
|
||||
appended ONLY when the subfolder's summary is stored), a blank
|
||||
line, the file lines in EXACTLY the existing
|
||||
``source: X | path: Y | title: Z`` format (the canonical
|
||||
``read``/``grep`` identity — unchanged), and the cap note
|
||||
line, the file lines in EXACTLY the
|
||||
``source: X | path: Y | title: Z | date: YYYY-MM-DD`` format (the
|
||||
canonical ``read``/``grep`` identity plus the phase-106 D5
|
||||
appended ``date`` field — the only changed part), and the cap note
|
||||
``…and {hidden} more documents in this folder — use grep
|
||||
(pattern) to find a specific one.`` ONLY when the folder's own
|
||||
files outnumber :data:`LS_MAX_FILE_LINES` (*files* arrives capped;
|
||||
@@ -891,8 +904,8 @@ def render_folder_listing(
|
||||
if files or total_files > len(files):
|
||||
body.append("")
|
||||
body.extend(
|
||||
f"source: {source} | path: {path} | title: {title}"
|
||||
for source, path, title in files
|
||||
f"source: {source} | path: {path} | title: {title} | date: {date}"
|
||||
for source, path, title, date in files
|
||||
)
|
||||
hidden = total_files - len(files)
|
||||
if hidden > 0:
|
||||
@@ -1172,15 +1185,26 @@ def _execute_tool(
|
||||
holder.read_truncations.append(
|
||||
(cast("str", raw_path), cap, len(doc.content))
|
||||
)
|
||||
# Phase 106 (D5): the date rides every document the model
|
||||
# sees — the ``read`` result's SECOND line; the FIRST line
|
||||
# stays ``Document {source}/{path}:`` BYTE-IDENTICAL (the
|
||||
# E2E mock's ``_READ_RESULT_PREFIX`` header contract).
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{doc.content[:cap]}\n"
|
||||
f"{TRUNCATION_MARKER}\n"
|
||||
f"{READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))}"
|
||||
)
|
||||
# At or under the cap: byte-identical to the pre-phase-95 result
|
||||
# (no marker, no notice, no holder entry, no ToolResultPiece).
|
||||
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
|
||||
# At or under the cap: the pre-phase-95 result plus the
|
||||
# phase-106 D5 date line (first line byte-identical — the
|
||||
# mock's header contract; no marker, no notice, no holder
|
||||
# entry, no ToolResultPiece).
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{doc.content}"
|
||||
)
|
||||
if call.name == "grep":
|
||||
raw_pattern = call.arguments.get("pattern")
|
||||
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
|
||||
|
||||
@@ -18,6 +18,13 @@ The guarantees (phase 49 locked decisions):
|
||||
unpack directory, and device/FIFO members — and counting every
|
||||
extracted byte against a cap (zip-bomb guard). Any failure removes the
|
||||
partial ``target_dir`` so no half-unpacked tree survives.
|
||||
* :func:`unpack_archive` also restores each regular file's member mtime
|
||||
(the zip DOS ``date_time`` or the tar ``mtime``) — phase 106, D2:
|
||||
uploaded archives keep their file dates, which the importer reads as
|
||||
the document creation date. Directories, symlinks, and hardlinks are
|
||||
untouched; the cap and every safety check above are unchanged (the
|
||||
``utime`` sits after a successful ``_write_capped``, so a failed
|
||||
unpack still removes the partial tree).
|
||||
* :func:`swap_in` makes ``new_dir`` become ``final_dir`` with **no
|
||||
missing window**: the previous folder is renamed to a unique
|
||||
same-filesystem ``.old-`` sibling first, the new folder is renamed
|
||||
@@ -33,6 +40,7 @@ import stat
|
||||
import tarfile
|
||||
import uuid
|
||||
import zipfile
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import IO
|
||||
|
||||
@@ -183,6 +191,16 @@ def _unpack_zip(archive: Path, target: Path, max_extract_bytes: int) -> None:
|
||||
else:
|
||||
with zf.open(member) as src:
|
||||
_write_capped(src, dest, max_extract_bytes, total)
|
||||
# Phase 106 (D2): restore the member's DOS mtime — a
|
||||
# tz-agnostic epoch value, UTC-rendered exactly like an
|
||||
# mtime — so uploaded archives keep their file dates.
|
||||
# Regular files only; an OSError here still removes the
|
||||
# partial tree in unpack_archive like any write failure.
|
||||
# ``ns=`` takes INTEGER nanoseconds (a float seconds
|
||||
# value raises TypeError), so convert explicitly.
|
||||
mtime = datetime(*member.date_time, tzinfo=UTC).timestamp()
|
||||
mtime_ns = int(mtime * 1_000_000_000)
|
||||
os.utime(dest, ns=(mtime_ns, mtime_ns))
|
||||
|
||||
|
||||
def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
|
||||
@@ -206,6 +224,14 @@ def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
|
||||
if src is None:
|
||||
raise ArchiveUploadError("corrupt archive member")
|
||||
_write_capped(src, dest, max_extract_bytes, total)
|
||||
# Phase 106 (D2): restore the member's mtime (epoch
|
||||
# seconds — a tz-agnostic value) so uploaded archives
|
||||
# keep their file dates. Regular files only; an OSError
|
||||
# here still removes the partial tree in unpack_archive.
|
||||
# ``ns=`` takes INTEGER nanoseconds (a float seconds
|
||||
# value raises TypeError), so convert explicitly.
|
||||
mtime_ns = int(member.mtime * 1_000_000_000)
|
||||
os.utime(dest, ns=(mtime_ns, mtime_ns))
|
||||
else: # char/block device, FIFO
|
||||
raise ArchiveUploadError(
|
||||
"tar archives with device or FIFO members are not allowed"
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Document-creation-date sourcing + normalization (phase 106, D2/D3).
|
||||
|
||||
Every document date the importer writes and every date the owner
|
||||
edits passes through :func:`normalize_doc_date` — the single choke
|
||||
point for the owner's rules: an UNDETERMINED date (no source signal)
|
||||
and a FUTURE date (beyond a small clock-skew tolerance) both assume
|
||||
the document was created TODAY (UTC). Naive source timestamps (zip
|
||||
DOS mtimes, tar mtimes, git-free fallbacks) are tz-agnostic epoch-
|
||||
based values rendered as UTC; aware ones are converted to UTC.
|
||||
|
||||
Pure and stdlib-only by contract (unit-pinned in
|
||||
``tests/unit/test_doc_dates.py``): no database, no logging, no I/O
|
||||
besides :func:`file_mtime_datetime`'s single ``stat`` — the callers
|
||||
(importer task 04, the date-edit API task 05) own everything else.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
__all__ = ["FUTURE_SKEW_TOLERANCE", "file_mtime_datetime", "normalize_doc_date"]
|
||||
|
||||
#: Clock-skew tolerance (D3): a source date up to this far in the
|
||||
#: FUTURE is a drifting clock, not a future document — it keeps its
|
||||
#: date. Beyond it, the owner's rule applies (→ today).
|
||||
FUTURE_SKEW_TOLERANCE = timedelta(days=1)
|
||||
|
||||
|
||||
def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime:
|
||||
"""*raw* → the stored UTC creation date (the D3 rule, pinned).
|
||||
|
||||
``now`` is injectable (tests); it defaults to
|
||||
``datetime.now(UTC)``. ``raw=None`` (undetermined) → *now*;
|
||||
naive *raw* → treated as UTC; aware *raw* → converted to UTC;
|
||||
*raw* beyond *now* + :data:`FUTURE_SKEW_TOLERANCE` → *now*.
|
||||
The result always carries full precision (no date-truncation —
|
||||
the display formats, the storage doesn't).
|
||||
"""
|
||||
if now is None:
|
||||
now = datetime.now(UTC)
|
||||
elif now.tzinfo is None:
|
||||
# The future check compares in AWARE space — a naive ``now``
|
||||
# (callers/tests) is a UTC instant, like the naive ``raw``.
|
||||
now = now.replace(tzinfo=UTC)
|
||||
if raw is None:
|
||||
return now
|
||||
# Epoch-based source values (zip DOS times, tar mtimes) are
|
||||
# tz-agnostic — attach UTC; never assume the host's local TZ.
|
||||
# Aware values are converted to UTC (the comparison below is
|
||||
# done in aware space).
|
||||
raw = raw.replace(tzinfo=UTC) if raw.tzinfo is None else raw.astimezone(UTC)
|
||||
if raw > now + FUTURE_SKEW_TOLERANCE:
|
||||
# Genuinely future (beyond the clock-skew tolerance) → today.
|
||||
return now
|
||||
return raw
|
||||
|
||||
|
||||
def file_mtime_datetime(path: Path) -> datetime:
|
||||
"""The file's mtime as an aware UTC datetime (the D2 fallback).
|
||||
|
||||
Epoch mtimes are tz-agnostic — UTC is the correct rendering
|
||||
(zip DOS timestamps and tar mtimes pass through the same
|
||||
:func:`normalize_doc_date` after unpacking, task 03).
|
||||
"""
|
||||
return datetime.fromtimestamp(path.stat().st_mtime, tz=UTC)
|
||||
+80
-3
@@ -31,6 +31,17 @@ previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
|
||||
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
|
||||
summary line with per-format counts (PLAN §9).
|
||||
|
||||
Document dates (phase 106, D2/D4): every import sources
|
||||
``documents.created_at`` from the file's source — the per-file git
|
||||
last-commit date when a ``doc_dates_by_root`` entry names the file,
|
||||
else the file's mtime — normalized by
|
||||
:func:`app.rag.doc_dates.normalize_doc_date` (undetermined or future →
|
||||
today, D3) on every add and update. On the unchanged path the stored
|
||||
date is REFRESHED from the same source (it may go OLDER — no monotonic
|
||||
guard) and counted in ``summary.dates_updated`` — unless the row
|
||||
carries the owner's manual correction (``created_at_manual``, D1), which
|
||||
the sync never touches.
|
||||
|
||||
``import_sources`` accepts an optional per-file ``progress`` callback
|
||||
(phase 64, task 01) reporting the file being processed right now.
|
||||
"""
|
||||
@@ -51,6 +62,7 @@ from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.chunker import chunk_document, extract_title
|
||||
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
|
||||
from app.rag.llm import EmbeddingError, LLMError
|
||||
from app.rag.summarizer import generate_summary
|
||||
|
||||
@@ -93,6 +105,12 @@ class ImportSummary:
|
||||
#: Non-markdown files whose summary generation failed (best-effort —
|
||||
#: the document is still indexed, without a summary).
|
||||
summary_errors: int = 0
|
||||
#: Files whose ``created_at`` was refreshed on the UNCHANGED path —
|
||||
#: content untouched, date re-sourced (phase 106, D4: the date may
|
||||
#: go OLDER; a date-only refresh NEVER counts added/updated/pruned,
|
||||
#: so no ``sources_meta`` bump, no overview/folder-summary
|
||||
#: regeneration).
|
||||
dates_updated: int = 0
|
||||
#: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
|
||||
formats: dict[str, int] = field(default_factory=dict)
|
||||
|
||||
@@ -107,7 +125,7 @@ class ImportSummary:
|
||||
logger.info(
|
||||
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
|
||||
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
|
||||
"formats=%s",
|
||||
"dates_updated=%d formats=%s",
|
||||
self.files,
|
||||
self.added,
|
||||
self.updated,
|
||||
@@ -118,6 +136,7 @@ class ImportSummary:
|
||||
self.embed_batches,
|
||||
self.summaries,
|
||||
self.summary_errors,
|
||||
self.dates_updated,
|
||||
self.format_counts(),
|
||||
)
|
||||
|
||||
@@ -248,6 +267,7 @@ async def import_sources(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
) -> ImportSummary:
|
||||
"""Import every A9-format file under *sources* (see module docstring).
|
||||
|
||||
@@ -288,6 +308,19 @@ async def import_sources(
|
||||
flag ON and is walked again with it OFF simply never enters
|
||||
``seen``, so the next ``prune=True`` run deletes its row
|
||||
automatically (A2 — the A9/phase-89 precedent).
|
||||
|
||||
``doc_dates_by_root`` (phase 106, D2/D4) maps ``str(root)`` — the
|
||||
root path string exactly as passed in *sources* — to that source's
|
||||
RAW per-file source dates: source-relative POSIX path → the git
|
||||
last-commit datetime (task 03's ``file_commit_dates``). ONLY git
|
||||
roots are listed — unlisted roots (local dirs, unpacked uploads)
|
||||
take the mtime fallback, and a path missing from its root's map
|
||||
does too. The map entry beats the file's mtime when present. The
|
||||
progress pre-walk is untouched (dates change no file count).
|
||||
``None`` (the default) changes nothing for existing callers: the
|
||||
mtime fallback applies to every file — which IS the behavior
|
||||
change, D4: an unchanged file now refreshes its stored date from
|
||||
its source on every run (the backfill-correction case).
|
||||
"""
|
||||
if limit is not None and limit <= 0:
|
||||
raise ValueError("limit must be >= 1")
|
||||
@@ -325,6 +358,10 @@ async def import_sources(
|
||||
source_names.add(source)
|
||||
ignore = _ignore_for_root(root, ignore_by_root)
|
||||
include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
|
||||
# Phase 106 (D2): the root's raw source dates (git last-commit
|
||||
# for git roots, keyed by the same str(root) convention); {}
|
||||
# for unlisted roots — every file then takes the mtime fallback.
|
||||
dates_map = (doc_dates_by_root or {}).get(str(root), {})
|
||||
for path in iter_importable_files(
|
||||
root,
|
||||
llm.settings.import_extension_set,
|
||||
@@ -353,7 +390,7 @@ async def import_sources(
|
||||
try:
|
||||
await _index_file(
|
||||
session, source=source, rel=rel, full_path=path, llm=llm,
|
||||
summary=summary,
|
||||
summary=summary, raw_date=dates_map.get(rel),
|
||||
)
|
||||
except EmbeddingError as e:
|
||||
# A pathological file (e.g. content the embedding endpoint
|
||||
@@ -384,15 +421,45 @@ async def _index_file(
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
) -> None:
|
||||
"""Upsert one file: doc row + chunk rows + embeddings, one transaction."""
|
||||
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
|
||||
|
||||
``raw_date`` (phase 106, D2) is the file's RAW source date — the
|
||||
git last-commit datetime from the caller's ``doc_dates_by_root``
|
||||
map, or ``None`` (every non-git case): the file's mtime is read
|
||||
here, once, and becomes the source date (the D2 fallback).
|
||||
"""
|
||||
settings = llm.settings
|
||||
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||||
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
if raw_date is None:
|
||||
# D2 fallback: no source date in the map → the file's mtime
|
||||
# (one stat). Read before the unchanged early-return — the
|
||||
# unchanged path refreshes the stored date from the same source.
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
summary.unchanged += 1
|
||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||
if doc.created_at_manual:
|
||||
# D1/D4: the owner's correction survives the sync — no
|
||||
# write at all (the phase-97 ``manually_edited`` precedent).
|
||||
return
|
||||
# D4: the date refreshes on every sync, including unchanged
|
||||
# files, and may go OLDER (no monotonic guard). A date-only
|
||||
# refresh is still counted ``unchanged`` — never added/updated/
|
||||
# pruned, so no ``sources_meta`` bump and no regeneration.
|
||||
target = normalize_doc_date(raw_date)
|
||||
if target != doc.created_at:
|
||||
doc.created_at = target
|
||||
session.commit()
|
||||
summary.dates_updated += 1
|
||||
logger.info(
|
||||
"import: date-refreshed source=%s path=%s date=%s",
|
||||
source, rel,
|
||||
doc.created_at.isoformat(),
|
||||
)
|
||||
return
|
||||
|
||||
verb = "updated" if doc is not None else "added"
|
||||
@@ -411,6 +478,11 @@ async def _index_file(
|
||||
content=content,
|
||||
content_hash=digest,
|
||||
indexed_at=datetime.now(UTC),
|
||||
# Phase 106 (D2/D3): the sourced creation date, normalized
|
||||
# (undetermined or future → today). ``created_at_manual``
|
||||
# stays the column default (False) — only the date-edit API
|
||||
# (task 05) sets it.
|
||||
created_at=normalize_doc_date(raw_date),
|
||||
)
|
||||
session.add(doc)
|
||||
else:
|
||||
@@ -419,6 +491,11 @@ async def _index_file(
|
||||
doc.content = content
|
||||
doc.content_hash = digest
|
||||
doc.indexed_at = datetime.now(UTC)
|
||||
# Phase 106 (D4): a content change is a new document version —
|
||||
# the date is re-sourced and a previous manual correction is
|
||||
# reset (it referred to the old content).
|
||||
doc.created_at = normalize_doc_date(raw_date)
|
||||
doc.created_at_manual = False
|
||||
|
||||
session.flush() # guarantees doc.id even for brand-new rows
|
||||
|
||||
|
||||
+12
-1
@@ -352,6 +352,12 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
|
||||
cap — not the prompt — decides whether the tools are actually
|
||||
offered to the model, see :mod:`app.rag.agent`).
|
||||
|
||||
Each ``<document>`` block carries the identity attributes
|
||||
``source`` / ``path`` / ``title`` — and, since phase 106 (D5),
|
||||
``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``,
|
||||
APPENDED after ``title`` — the only position; always present,
|
||||
``created_at`` is NOT NULL) — plus the document's full text.
|
||||
|
||||
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
|
||||
LEADING this section (the document texts are already context — do
|
||||
not ``read`` one the user asked to open) was tried and REVERTED:
|
||||
@@ -368,8 +374,13 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
|
||||
# attribute at the ``source``/``path`` copy site was TRIED and
|
||||
# REVERTED the same day (no improvement across runs; the block stays
|
||||
# exactly the document identity + full text).
|
||||
# Phase 106 (D5): every document the model sees carries its
|
||||
# creation date — the block's ``date`` attribute (the row's
|
||||
# ``created_at`` UTC date part, appended after ``title`` — the
|
||||
# only position; always present, ``created_at`` is NOT NULL).
|
||||
blocks = [
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}">\n'
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
|
||||
f'date="{doc.created_at:%Y-%m-%d}">\n'
|
||||
f"{doc.content}\n"
|
||||
"</document>"
|
||||
for doc in documents
|
||||
|
||||
+81
-1
@@ -41,10 +41,12 @@ Deterministic tie-break for equal fused scores:
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
import uuid
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, replace
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select, text
|
||||
@@ -146,6 +148,7 @@ _LEXICAL_SQL = text(
|
||||
d.content AS doc_content,
|
||||
d.content_hash AS content_hash,
|
||||
d.indexed_at AS indexed_at,
|
||||
d.created_at AS created_at,
|
||||
c.is_summary AS is_summary,
|
||||
ts_rank(c.tsv, to_tsquery('english', :tsquery)) AS rank
|
||||
FROM chunks c
|
||||
@@ -242,6 +245,73 @@ def fuse(
|
||||
return out
|
||||
|
||||
|
||||
def apply_recency_boost(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
*,
|
||||
now: datetime | None = None,
|
||||
weight: float | None = None,
|
||||
half_life_days: int | None = None,
|
||||
) -> list[RetrievedChunk]:
|
||||
"""Additive recency boost on the fused score (phase 106, D6).
|
||||
|
||||
Each chunk's score becomes
|
||||
``score + weight * exp(−age_days / half_life_days)`` where
|
||||
``age_days = max(0, (now − document.created_at))`` in days — a
|
||||
zero-age document gets the full *weight* (the MAXIMUM additive
|
||||
score), each ``half_life_days`` of age multiplies the remaining
|
||||
boost by ``e**-1`` (≈0.37), and a FUTURE ``created_at`` clamps to
|
||||
age 0 (the document reads as brand-new — consistent with D3's
|
||||
today-folding in :mod:`app.rag.doc_dates`). Defaults: *weight* /
|
||||
*half_life_days* from :func:`get_settings` (``recency_boost`` /
|
||||
``recency_half_life_days``), *now* from ``datetime.now(UTC)``.
|
||||
|
||||
Magnitude rationale (the ``0.0007`` default, the k=60 RRF scale):
|
||||
rank 1 vs 2 in one list differs by ~0.00026 and rank 1 vs 10 by
|
||||
~0.0021, so the full weight is a bounded 2-3 rank head start —
|
||||
enough to break near-ties toward the newer document, far below the
|
||||
fused gap between a document that answers and one that merely
|
||||
resembles (the phase-106 fine-line battery pins the measured
|
||||
margin: 0.00263 ≥ 3× the zero-age boost).
|
||||
|
||||
Pure (the :func:`fuse` convention): the inputs are never mutated —
|
||||
every boosted chunk is a ``replace()`` copy — and the result is
|
||||
re-sorted with the EXISTING deterministic key
|
||||
``(−score, −cosine, document.path, position)``; with ``weight=0``
|
||||
every score is untouched and an already-fused (already-sorted)
|
||||
input comes back byte-identical (the kill switch, pinned).
|
||||
|
||||
Untouched by design: the A8 honesty gate and ``query_log.top_score``
|
||||
(both read the chunk's ``cosine``, which the boost never modifies),
|
||||
:func:`weak_hit_titles` (titles only), and the never-truncated
|
||||
top-N contract (:func:`select_documents` still feeds whole
|
||||
documents — the boost re-ranks WHICH documents, never truncates).
|
||||
SINGLE APPLY SITE: :func:`retrieve()` is the only caller in
|
||||
``app/`` — the chat API and ``scripts/eval_retrieval.py`` inherit
|
||||
the boost through it; nothing else may apply it.
|
||||
"""
|
||||
if weight is None or half_life_days is None:
|
||||
settings = get_settings()
|
||||
if weight is None:
|
||||
weight = settings.recency_boost
|
||||
if half_life_days is None:
|
||||
half_life_days = settings.recency_half_life_days
|
||||
if half_life_days <= 0:
|
||||
raise ValueError("half_life_days must be > 0")
|
||||
if now is None:
|
||||
now = datetime.now(UTC)
|
||||
boosted: list[RetrievedChunk] = []
|
||||
for rc in chunks:
|
||||
age_days = max(0.0, (now - rc.document.created_at).total_seconds() / 86400.0)
|
||||
boosted.append(
|
||||
replace(
|
||||
rc,
|
||||
score=rc.score + weight * math.exp(-age_days / half_life_days),
|
||||
)
|
||||
)
|
||||
boosted.sort(key=lambda rc: (-rc.score, -rc.cosine, rc.document.path, rc.position))
|
||||
return boosted
|
||||
|
||||
|
||||
def _vector_candidates(
|
||||
db: Session, question_embedding: list[float], limit: int
|
||||
) -> list[RetrievedChunk]:
|
||||
@@ -289,6 +359,7 @@ _NAME_HIT_SQL = text(
|
||||
d.content AS doc_content,
|
||||
d.content_hash AS content_hash,
|
||||
d.indexed_at AS indexed_at,
|
||||
d.created_at AS created_at,
|
||||
c.id AS chunk_id,
|
||||
c.position AS position,
|
||||
c.content AS content,
|
||||
@@ -364,6 +435,7 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]:
|
||||
content=row.doc_content,
|
||||
content_hash=row.content_hash,
|
||||
indexed_at=row.indexed_at,
|
||||
created_at=row.created_at,
|
||||
)
|
||||
out.append(
|
||||
RetrievedChunk(
|
||||
@@ -418,6 +490,7 @@ def _lexical_candidates(db: Session, question: str, limit: int) -> list[Retrieve
|
||||
content=row.doc_content,
|
||||
content_hash=row.content_hash,
|
||||
indexed_at=row.indexed_at,
|
||||
created_at=row.created_at,
|
||||
)
|
||||
out.append(
|
||||
RetrievedChunk(
|
||||
@@ -460,7 +533,14 @@ def retrieve(
|
||||
raise ValueError("lexical_candidates must be >= 1")
|
||||
vector = _vector_candidates(db, question_embedding, v_n)
|
||||
lexical = _lexical_candidates(db, question, l_n)
|
||||
return fuse(vector, lexical, settings.rrf_k)
|
||||
fused = fuse(vector, lexical, settings.rrf_k)
|
||||
if settings.recency_boost > 0:
|
||||
# Phase 106, D6 — the SINGLE recency-boost apply site: an
|
||||
# additive post-fusion re-rank (see :func:`apply_recency_boost`).
|
||||
# ``0`` = off: the pre-phase ranking returns byte-identical (the
|
||||
# kill switch) and weight-0 callers pay nothing.
|
||||
return apply_recency_boost(fused)
|
||||
return fused
|
||||
|
||||
|
||||
def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]:
|
||||
|
||||
Reference in New Issue
Block a user