phase: 106_document_dates
Everything is verified green. Final report: **Phase 106 — Document dates (final verification pass; all 10 tasks already complete)** - Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed. - `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓) - `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up) - 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation** - `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings** **Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions. - **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring). - **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+45
-22
@@ -33,7 +33,12 @@ with no ignore. Phase 105 extends the same resolution with each row's
|
||||
``include_hidden`` flag — a second per-root map keyed by the same root
|
||||
strings (the importer reads it per root); manual ``--source`` dirs and
|
||||
the legacy fallback have no rows, so they import with the empty map
|
||||
(hidden paths skipped — A4).
|
||||
(hidden paths skipped — A4). Phase 106 (D2) extends it a third time
|
||||
with each GIT row's per-file last-commit dates — a map keyed by the
|
||||
same root strings, built from ``file_commit_dates`` after the clone;
|
||||
manual ``--source`` dirs and the legacy fallback have no rows (no
|
||||
clone), so they import with the empty map and take the importer's
|
||||
mtime fallback.
|
||||
|
||||
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
|
||||
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
|
||||
@@ -97,6 +102,7 @@ import asyncio
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
@@ -110,7 +116,7 @@ from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.overview import regenerate_overview
|
||||
from app.rag.sources_meta import bump_sources_version
|
||||
from scripts.git_sync import GitSyncError, clone_or_pull
|
||||
from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates
|
||||
|
||||
logger = logging.getLogger("scripts.import_docs")
|
||||
|
||||
@@ -166,11 +172,16 @@ def repo_name(url: str) -> str:
|
||||
|
||||
|
||||
def _resolve_sources(
|
||||
cli_sources: list[Path] | None, settings: Settings
|
||||
) -> tuple[list[Path], dict[str, list[str]], dict[str, bool]]:
|
||||
cli_sources: list[Path] | None, settings: Settings,
|
||||
) -> tuple[
|
||||
list[Path],
|
||||
dict[str, list[str]],
|
||||
dict[str, bool],
|
||||
dict[str, dict[str, datetime]],
|
||||
]:
|
||||
"""Resolve the directories to import (phase 28, extended in phases
|
||||
35 and 38; per-root ignore maps, phase 89; per-root hidden-folders
|
||||
flag maps, phase 105).
|
||||
flag maps, phase 105; per-root source-date maps, phase 106).
|
||||
|
||||
Precedence: ``--source`` (explicit manual paths — always wins) >
|
||||
the effective sources — the ``git_sources`` DB rows (git + local),
|
||||
@@ -183,23 +194,28 @@ def _resolve_sources(
|
||||
stored directory, re-verified ``.is_dir()`` at run time) > the
|
||||
legacy ``DEFAULT_SOURCES``.
|
||||
|
||||
Returns ``(sources, ignore_by_root, include_hidden_by_root)``
|
||||
(phase 89; phase 105 adds the per-root flag map — the flag is
|
||||
stored per row, manual ``--source`` dirs and the legacy fallback
|
||||
have no rows and import with the empty map: hidden paths skipped,
|
||||
A4): both maps are keyed by the resolved root string, exactly as
|
||||
Returns ``(sources, ignore_by_root, include_hidden_by_root,
|
||||
doc_dates_by_root)`` (phase 89; phase 105 adds the per-root flag
|
||||
map — the flag is stored per row, manual ``--source`` dirs and the
|
||||
legacy fallback have no rows and import with the empty map: hidden
|
||||
paths skipped, A4; phase 106 adds the per-root source-date map):
|
||||
all three maps are keyed by the resolved root string, exactly as
|
||||
the importer sees it (two rows sharing a root string get the union
|
||||
— extend, not replace — for the ignore lists, and the OR of their
|
||||
flags for the hidden map); manual ``--source`` dirs and the legacy
|
||||
fallback have no rows, so they import with empty maps (no ignore,
|
||||
hidden skipped).
|
||||
— extend, not replace — for the ignore lists, the OR of their
|
||||
flags for the hidden map, and one date walk for the date map);
|
||||
the date map lists ONLY git roots (the ``file_commit_dates`` walk
|
||||
over the fresh checkout after the clone — fail-soft to ``{}``,
|
||||
which the importer reads as "no source dates, use mtimes"), and
|
||||
manual ``--source`` dirs and the legacy fallback have no rows, so
|
||||
they import with empty maps (no ignore, hidden skipped, mtime
|
||||
fallback).
|
||||
|
||||
A :class:`GitSyncError` from a failing clone/pull — or a missing
|
||||
local directory (``local source missing: <path>``) — propagates to
|
||||
:func:`main`, which aborts the run before importing anything.
|
||||
"""
|
||||
if cli_sources:
|
||||
return [path.expanduser() for path in cli_sources], {}, {}
|
||||
return [path.expanduser() for path in cli_sources], {}, {}, {}
|
||||
db = SessionLocal()
|
||||
try:
|
||||
rows, origin = effective_sources(db)
|
||||
@@ -215,9 +231,14 @@ def _resolve_sources(
|
||||
sources: list[Path] = []
|
||||
ignore_by_root: dict[str, list[str]] = {}
|
||||
include_hidden_by_root: dict[str, bool] = {}
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] = {}
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
|
||||
# Phase 106 (D2): the checkout's per-file last-commit
|
||||
# dates, keyed by the SAME root string the importer
|
||||
# sees; local rows contribute nothing (mtime fallback).
|
||||
doc_dates_by_root[str(root)] = file_commit_dates(root)
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
@@ -242,8 +263,8 @@ def _resolve_sources(
|
||||
include_hidden_by_root.get(str(root), False)
|
||||
or bool(row.include_hidden)
|
||||
)
|
||||
return sources, ignore_by_root, include_hidden_by_root
|
||||
return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}
|
||||
return sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root
|
||||
return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}, {}
|
||||
|
||||
|
||||
def _overview_row_exists() -> bool:
|
||||
@@ -283,12 +304,13 @@ def main(argv: list[str] | None = None) -> int:
|
||||
# Git sources resolve (and clone/pull) *before* any import: a failing
|
||||
# repo aborts the run with a non-zero exit, naming the failure — a bad
|
||||
# URL must never silently import partial junk. The second element is
|
||||
# the phase-89 per-root ignore map and the third the phase-105
|
||||
# per-root hidden-folders flag map (both empty for manual/fallback
|
||||
# paths).
|
||||
# the phase-89 per-root ignore map, the third the phase-105
|
||||
# per-root hidden-folders flag map, and the fourth the phase-106
|
||||
# per-root source-date map (git rows only — empty for manual/
|
||||
# fallback paths, which take the importer's mtime fallback).
|
||||
try:
|
||||
sources, ignore_by_root, include_hidden_by_root = _resolve_sources(
|
||||
args.source, settings
|
||||
sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root = (
|
||||
_resolve_sources(args.source, settings)
|
||||
)
|
||||
except GitSyncError as e:
|
||||
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
|
||||
@@ -356,6 +378,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
sources, llm, prune=args.prune, limit=args.limit,
|
||||
ignore_by_root=ignore_by_root,
|
||||
include_hidden_by_root=include_hidden_by_root,
|
||||
doc_dates_by_root=doc_dates_by_root,
|
||||
)
|
||||
if args.limit is not None:
|
||||
# An incomplete walk is debug-only — it must never advance
|
||||
|
||||
Reference in New Issue
Block a user