phase: 106_document_dates
Everything is verified green. Final report: **Phase 106 — Document dates (final verification pass; all 10 tasks already complete)** - Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed. - `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓) - `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up) - 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation** - `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings** **Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions. - **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring). - **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+99
-2
@@ -11,15 +11,33 @@ used.
|
||||
|
||||
This module is the only place the ``git`` CLI is invoked (A11: stdlib
|
||||
``subprocess`` only, no new packages) — every git command goes through
|
||||
:func:`run_git`: the clone/pull in :func:`clone_or_pull` and the
|
||||
:func:`run_git`: the clone/pull in :func:`clone_or_pull`, the per-file
|
||||
last-commit-date walk in :func:`file_commit_dates` (phase 106), and the
|
||||
docs-push sequence in :mod:`app.core.docs_push` (phase 59).
|
||||
|
||||
Per-file last-commit dates (phase 106, D2/D10) — behavior verified
|
||||
against scratch repos 2026-09-13:
|
||||
|
||||
* a LOCAL-PATH checkout made by :func:`clone_or_pull` keeps FULL
|
||||
history (``git clone --depth 1 /local/path`` prints "--depth is
|
||||
ignored in local clones" and does not shallow) →
|
||||
:func:`file_commit_dates` yields TRUE per-file last-commit dates;
|
||||
* a URL-TRANSPORT checkout (https/ssh/``file://``) is shallow, and in a
|
||||
shallow clone git reports the TIP commit as every existing file's
|
||||
last commit (the shallow boundary is each file's history root) → a
|
||||
uniform per-repo tip date: no intra-repo distortion, a real
|
||||
cross-source signal, refreshed on every pull.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
__all__ = ["GitSyncError", "clone_or_pull", "run_git"]
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
__all__ = ["GitSyncError", "clone_or_pull", "file_commit_dates", "run_git"]
|
||||
|
||||
|
||||
class GitSyncError(RuntimeError):
|
||||
@@ -48,6 +66,85 @@ def clone_or_pull(url: str, dest: Path | str) -> Path:
|
||||
return dest
|
||||
|
||||
|
||||
def _parse_commit_dates(output: str) -> dict[str, datetime]:
|
||||
"""Parse ``git log --name-only --format=@@%cI`` output (newest first).
|
||||
|
||||
A ``@@<ISO-8601>`` line starts a commit (``%cI`` is ISO-strict, so
|
||||
the date is always aware — parsed with ``datetime.fromisoformat``);
|
||||
the following non-empty, non-``@@`` lines are repo-relative paths.
|
||||
The FIRST sighting of a path wins (the walk is newest-first) — that
|
||||
is the file's last-commit date. Paths are split on whitespace (like
|
||||
name-only output), ``\\``-normalized to ``/``, and a leading ``/``
|
||||
is stripped. Raises ``ValueError`` on a malformed commit date or a
|
||||
path line before any commit header (the caller fails soft).
|
||||
"""
|
||||
dates: dict[str, datetime] = {}
|
||||
commit: datetime | None = None
|
||||
for line in output.splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
if line.startswith("@@"):
|
||||
commit = datetime.fromisoformat(line[2:])
|
||||
continue
|
||||
if commit is None:
|
||||
raise ValueError(f"path line before any commit header: {line!r}")
|
||||
for raw_path in line.split():
|
||||
path = raw_path.replace("\\", "/").lstrip("/")
|
||||
if path:
|
||||
dates.setdefault(path, commit)
|
||||
return dates
|
||||
|
||||
|
||||
def file_commit_dates(dest: Path | str) -> dict[str, datetime]:
|
||||
"""Per-file last-commit dates for one checkout (phase 106, D2).
|
||||
|
||||
ONE ``git log --name-only --format=@@%cI`` walk through
|
||||
:func:`run_git` (the A11 single-invocation site, one git call per
|
||||
source per sync) → ``{repo-relative POSIX path: last-commit
|
||||
datetime}``, newest-first so the first sighting of a path wins.
|
||||
|
||||
The checkout behavior is pinned (verified 2026-09-13 — see the
|
||||
module docstring): a local-path ``clone_or_pull`` checkout keeps
|
||||
FULL history → TRUE per-file dates; a URL-transport checkout is
|
||||
shallow → the repo's TIP-commit date for every working-tree file
|
||||
(D10: uniform within the repo, real across sources).
|
||||
|
||||
Fail-soft (pinned): a missing/non-directory checkout, a git failure
|
||||
(:class:`GitSyncError`), or ANY parse anomaly logs a warning and
|
||||
returns ``{}`` — the importer falls back to file mtimes; a date
|
||||
walk must never break a sync.
|
||||
"""
|
||||
dest = Path(dest)
|
||||
if not dest.is_dir():
|
||||
logger.warning(
|
||||
"file_commit_dates: %s is not a directory — no git dates "
|
||||
"(the importer will fall back to file mtimes)",
|
||||
dest,
|
||||
)
|
||||
return {}
|
||||
try:
|
||||
output = run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)
|
||||
except GitSyncError as exc:
|
||||
logger.warning(
|
||||
"file_commit_dates: git log failed for %s: %s — the importer "
|
||||
"will fall back to file mtimes",
|
||||
dest,
|
||||
exc,
|
||||
)
|
||||
return {}
|
||||
try:
|
||||
return _parse_commit_dates(output)
|
||||
except ValueError as exc:
|
||||
logger.warning(
|
||||
"file_commit_dates: unparseable git log output for %s (%s) — the "
|
||||
"importer will fall back to file mtimes",
|
||||
dest,
|
||||
exc,
|
||||
)
|
||||
return {}
|
||||
|
||||
|
||||
def run_git(argv: list[str], cwd: Path) -> str:
|
||||
"""Run one git command, capturing output; raise GitSyncError on failure.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user