phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s

Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-13 19:28:05 -04:00
parent cec819743d
commit ee3efb28c9
113 changed files with 8228 additions and 344 deletions
+45 -22
View File
@@ -33,7 +33,12 @@ with no ignore. Phase 105 extends the same resolution with each row's
``include_hidden`` flag — a second per-root map keyed by the same root
strings (the importer reads it per root); manual ``--source`` dirs and
the legacy fallback have no rows, so they import with the empty map
(hidden paths skipped — A4).
(hidden paths skipped — A4). Phase 106 (D2) extends it a third time
with each GIT row's per-file last-commit dates — a map keyed by the
same root strings, built from ``file_commit_dates`` after the clone;
manual ``--source`` dirs and the legacy fallback have no rows (no
clone), so they import with the empty map and take the importer's
mtime fallback.
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
@@ -97,6 +102,7 @@ import asyncio
import logging
import re
import sys
from datetime import datetime
from pathlib import Path
from app.config import Settings, get_settings
@@ -110,7 +116,7 @@ from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
from app.rag.sources_meta import bump_sources_version
from scripts.git_sync import GitSyncError, clone_or_pull
from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates
logger = logging.getLogger("scripts.import_docs")
@@ -166,11 +172,16 @@ def repo_name(url: str) -> str:
def _resolve_sources(
cli_sources: list[Path] | None, settings: Settings
) -> tuple[list[Path], dict[str, list[str]], dict[str, bool]]:
cli_sources: list[Path] | None, settings: Settings,
) -> tuple[
list[Path],
dict[str, list[str]],
dict[str, bool],
dict[str, dict[str, datetime]],
]:
"""Resolve the directories to import (phase 28, extended in phases
35 and 38; per-root ignore maps, phase 89; per-root hidden-folders
flag maps, phase 105).
flag maps, phase 105; per-root source-date maps, phase 106).
Precedence: ``--source`` (explicit manual paths — always wins) >
the effective sources — the ``git_sources`` DB rows (git + local),
@@ -183,23 +194,28 @@ def _resolve_sources(
stored directory, re-verified ``.is_dir()`` at run time) > the
legacy ``DEFAULT_SOURCES``.
Returns ``(sources, ignore_by_root, include_hidden_by_root)``
(phase 89; phase 105 adds the per-root flag map — the flag is
stored per row, manual ``--source`` dirs and the legacy fallback
have no rows and import with the empty map: hidden paths skipped,
A4): both maps are keyed by the resolved root string, exactly as
Returns ``(sources, ignore_by_root, include_hidden_by_root,
doc_dates_by_root)`` (phase 89; phase 105 adds the per-root flag
map — the flag is stored per row, manual ``--source`` dirs and the
legacy fallback have no rows and import with the empty map: hidden
paths skipped, A4; phase 106 adds the per-root source-date map):
all three maps are keyed by the resolved root string, exactly as
the importer sees it (two rows sharing a root string get the union
— extend, not replace — for the ignore lists, and the OR of their
flags for the hidden map); manual ``--source`` dirs and the legacy
fallback have no rows, so they import with empty maps (no ignore,
hidden skipped).
— extend, not replace — for the ignore lists, the OR of their
flags for the hidden map, and one date walk for the date map);
the date map lists ONLY git roots (the ``file_commit_dates`` walk
over the fresh checkout after the clone — fail-soft to ``{}``,
which the importer reads as "no source dates, use mtimes"), and
manual ``--source`` dirs and the legacy fallback have no rows, so
they import with empty maps (no ignore, hidden skipped, mtime
fallback).
A :class:`GitSyncError` from a failing clone/pull — or a missing
local directory (``local source missing: <path>``) — propagates to
:func:`main`, which aborts the run before importing anything.
"""
if cli_sources:
return [path.expanduser() for path in cli_sources], {}, {}
return [path.expanduser() for path in cli_sources], {}, {}, {}
db = SessionLocal()
try:
rows, origin = effective_sources(db)
@@ -215,9 +231,14 @@ def _resolve_sources(
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
include_hidden_by_root: dict[str, bool] = {}
doc_dates_by_root: dict[str, dict[str, datetime]] = {}
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
# Phase 106 (D2): the checkout's per-file last-commit
# dates, keyed by the SAME root string the importer
# sees; local rows contribute nothing (mtime fallback).
doc_dates_by_root[str(root)] = file_commit_dates(root)
else:
# kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the
@@ -242,8 +263,8 @@ def _resolve_sources(
include_hidden_by_root.get(str(root), False)
or bool(row.include_hidden)
)
return sources, ignore_by_root, include_hidden_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}
return sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}, {}
def _overview_row_exists() -> bool:
@@ -283,12 +304,13 @@ def main(argv: list[str] | None = None) -> int:
# Git sources resolve (and clone/pull) *before* any import: a failing
# repo aborts the run with a non-zero exit, naming the failure — a bad
# URL must never silently import partial junk. The second element is
# the phase-89 per-root ignore map and the third the phase-105
# per-root hidden-folders flag map (both empty for manual/fallback
# paths).
# the phase-89 per-root ignore map, the third the phase-105
# per-root hidden-folders flag map, and the fourth the phase-106
# per-root source-date map (git rows only — empty for manual/
# fallback paths, which take the importer's mtime fallback).
try:
sources, ignore_by_root, include_hidden_by_root = _resolve_sources(
args.source, settings
sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root = (
_resolve_sources(args.source, settings)
)
except GitSyncError as e:
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
@@ -356,6 +378,7 @@ def main(argv: list[str] | None = None) -> int:
sources, llm, prune=args.prune, limit=args.limit,
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
)
if args.limit is not None:
# An incomplete walk is debug-only — it must never advance