phase: 107_git_full_history_dates
All green — verification complete. Final report: **Phase 107 — final verification pass (all tasks already in `complete/`)** - **Verified:** `clone_or_pull` full-history fix (no `--depth` clone; `--is-shallow-repository` probe → one-time `git fetch --unshallow` self-heal → `git pull --ff-only`; D12 fail-loud) in `scripts/git_sync.py`; E2E suite `tests/e2e/test_git_source_dates.py` present; all five stale shallow/tip-date narratives corrected (no residual claims repo-wide; `docs_push.py` `--depth` fetches untouched as designed). - **Unit+integration:** `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v --no-cov` → **66 passed** (D10 tip-date test replaced by `test_url_clone_yields_true_per_file_dates` + `test_existing_shallow_checkout_self_heals`; unit pins: clone argv w/o `--depth`, probe→unshallow→pull order, unshallow-failure → `GitSyncError`). - **New E2E in isolation:** `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` → **4 passed** (old file 2020-06-15 ≠ tip file 2024-06-15 in `GET /api/docs`, Sources `Created`/`Updated` columns, viewer `Created` badge ISO title; folder `updated_at` subtree maxes). - **Phase-106 regressions in isolation:** `test_document_dates.py` → 6 passed; `test_sync_button.py` → 3 passed; `test_git_sources_admin.py` → 6 passed. - **Full gates:** `uv run pytest` → **2302 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check . && uv run pyright` → **clean, 0 errors**. - **Completion criteria:** 1 ✅ non-shallow URL clone + true per-file dates · 2 ✅ self-heal / plain-pull / fail-loud · 3 ✅ both suites green, bug-pin replaced · 4 ✅ E2E green in isolation · 5 ✅ all regressions + full suite + coverage + lint · 6 ✅ narratives corrected · 7 — no commit made (harness override: changes left in working tree; task files already in `complete/`). - **No defects found; no deviations.** Next pending phase: **108_history_wire_check**.
This commit is contained in:
+61
-30
@@ -1,8 +1,9 @@
|
||||
"""Git source sync for import_docs (phase 28).
|
||||
|
||||
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
|
||||
the first time, or fast-forwards an existing checkout with ``git pull
|
||||
--ff-only`` on subsequent runs.
|
||||
clone_or_pull(url, dest) clones ``url`` into ``dest`` with FULL history
|
||||
the first time, or — for an existing checkout — unshallows a shallow
|
||||
one first (``git fetch --unshallow``, the one-time self-heal) and then
|
||||
fast-forwards it with ``git pull --ff-only``.
|
||||
|
||||
Auth: nothing special — an ``https://…`` URL uses the OS credential
|
||||
helper / prompts; a ``git@host:repo.git`` URL uses the machine's SSH key.
|
||||
@@ -11,22 +12,34 @@ used.
|
||||
|
||||
This module is the only place the ``git`` CLI is invoked (A11: stdlib
|
||||
``subprocess`` only, no new packages) — every git command goes through
|
||||
:func:`run_git`: the clone/pull in :func:`clone_or_pull`, the per-file
|
||||
last-commit-date walk in :func:`file_commit_dates` (phase 106), and the
|
||||
docs-push sequence in :mod:`app.core.docs_push` (phase 59).
|
||||
:func:`run_git`: the clone, the shallow probe, the one-time
|
||||
``git fetch --unshallow``, and the ``git pull --ff-only`` in
|
||||
:func:`clone_or_pull`, the per-file last-commit-date walk in
|
||||
:func:`file_commit_dates` (phase 106), and the docs-push sequence in
|
||||
:mod:`app.core.docs_push` (phase 59).
|
||||
|
||||
Per-file last-commit dates (phase 106, D2/D10) — behavior verified
|
||||
against scratch repos 2026-09-13:
|
||||
Per-file last-commit dates (phase 106 D2; phase 107 D11 supersedes
|
||||
phase 106 D10) — behavior verified against scratch and live repos
|
||||
2026-09-13 / 2026-09-16:
|
||||
|
||||
* a LOCAL-PATH checkout made by :func:`clone_or_pull` keeps FULL
|
||||
history (``git clone --depth 1 /local/path`` prints "--depth is
|
||||
ignored in local clones" and does not shallow) →
|
||||
:func:`file_commit_dates` yields TRUE per-file last-commit dates;
|
||||
* a URL-TRANSPORT checkout (https/ssh/``file://``) is shallow, and in a
|
||||
shallow clone git reports the TIP commit as every existing file's
|
||||
last commit (the shallow boundary is each file's history root) → a
|
||||
uniform per-repo tip date: no intra-repo distortion, a real
|
||||
cross-source signal, refreshed on every pull.
|
||||
* every :func:`clone_or_pull` checkout is FULL history for EVERY
|
||||
transport (https/ssh/``file://``/local-path): a fresh clone carries
|
||||
no ``--depth`` (D11 — D10's ``--depth 1`` shallow URL clones are
|
||||
gone), and an existing shallow checkout (made pre-phase 107 — i.e.
|
||||
every deployed one, live + dev included) is probed with ``git
|
||||
rev-parse --is-shallow-repository`` and, while shallow, self-healed
|
||||
with the ONE-TIME ``git fetch --unshallow`` before the usual ``git
|
||||
pull --ff-only`` — no re-clone;
|
||||
* on a full-history checkout :func:`file_commit_dates` yields the
|
||||
TRUE per-file last-commit date for ALL git sources. The old
|
||||
shallow-clone behavior was the bug the owner reported 2026-09-16:
|
||||
in a shallow clone git reports the TIP commit as every existing
|
||||
file's last commit (the shallow boundary is each file's history
|
||||
root) → a uniform per-repo tip date (``container_bifrost`` shown as
|
||||
created on the repo tip, months off); after ``git fetch
|
||||
--unshallow`` on the live homelab checkout (438 commits visible)
|
||||
the walk returned the true dates — e.g. ``bifrost.md`` 2026-05-05,
|
||||
not the 2026-09-07 tip.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -45,23 +58,40 @@ class GitSyncError(RuntimeError):
|
||||
|
||||
|
||||
def clone_or_pull(url: str, dest: Path | str) -> Path:
|
||||
"""Clone ``url`` into ``dest`` (shallow, first run) or fast-forward it.
|
||||
"""Clone ``url`` into ``dest`` (full history, first run) or fast-forward it.
|
||||
|
||||
- dest without a ``.git`` (or absent) → ``git clone --depth 1 url dest``
|
||||
(shallow: the KB is re-imported incrementally anyway).
|
||||
- dest with a ``.git`` → ``git pull --ff-only`` (refuses to merge
|
||||
unrelated histories — a broken checkout fails loudly rather than
|
||||
producing a dirty index).
|
||||
- dest without a ``.git`` (or absent) → ``git clone url dest`` (full
|
||||
history for every transport — phase 107 D11: no ``--depth``, so a
|
||||
URL-transport checkout carries the whole commit log, not just the
|
||||
tip).
|
||||
- dest with a ``.git`` → probe ``git rev-parse
|
||||
--is-shallow-repository``; while shallow, ``git fetch
|
||||
--unshallow`` (the ONE-TIME self-heal for checkouts made shallow
|
||||
pre-phase 107 — the next sync of a deployed checkout becomes
|
||||
full-history without a re-clone), then ``git pull --ff-only``
|
||||
(refuses to merge unrelated histories — a broken checkout fails
|
||||
loudly rather than producing a dirty index).
|
||||
|
||||
Returns the destination path. Raises :class:`GitSyncError` when git is
|
||||
missing or a git invocation exits non-zero (with git's stderr in the
|
||||
message, so the caller can name the failing repo + reason).
|
||||
message, so the caller can name the failing repo + reason). D12
|
||||
fail-loud: a failed probe/unshallow/pull propagates exactly like a
|
||||
clone failure — never a silent fallback to tip dates or mtimes (a
|
||||
continued shallow checkout would silently re-serve the uniform tip
|
||||
date, i.e. the bug D11 fixes).
|
||||
"""
|
||||
dest = Path(dest)
|
||||
if not dest.exists() or not (dest / ".git").exists():
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
run_git(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
|
||||
run_git(["git", "clone", url, str(dest)], cwd=dest.parent)
|
||||
else:
|
||||
shallow = (
|
||||
run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest)
|
||||
.strip()
|
||||
== "true"
|
||||
)
|
||||
if shallow:
|
||||
run_git(["git", "fetch", "--unshallow"], cwd=dest)
|
||||
run_git(["git", "pull", "--ff-only"], cwd=dest)
|
||||
return dest
|
||||
|
||||
@@ -104,11 +134,12 @@ def file_commit_dates(dest: Path | str) -> dict[str, datetime]:
|
||||
source per sync) → ``{repo-relative POSIX path: last-commit
|
||||
datetime}``, newest-first so the first sighting of a path wins.
|
||||
|
||||
The checkout behavior is pinned (verified 2026-09-13 — see the
|
||||
module docstring): a local-path ``clone_or_pull`` checkout keeps
|
||||
FULL history → TRUE per-file dates; a URL-transport checkout is
|
||||
shallow → the repo's TIP-commit date for every working-tree file
|
||||
(D10: uniform within the repo, real across sources).
|
||||
Every ``clone_or_pull`` checkout is FULL history (fresh: no
|
||||
``--depth``; an existing shallow checkout is unshallowed on its
|
||||
next sync — phase 107 D11 supersedes phase 106 D10's shallow
|
||||
tip-date behavior) → TRUE per-file last-commit dates for ALL git
|
||||
sources, local AND URL (verified 2026-09-16 — see the module
|
||||
docstring).
|
||||
|
||||
Fail-soft (pinned): a missing/non-directory checkout, a git failure
|
||||
(:class:`GitSyncError`), or ANY parse anomaly logs a warning and
|
||||
|
||||
@@ -16,7 +16,8 @@ precedence order:
|
||||
(both kinds), else the ``BOR_GIT_SOURCES`` (comma-separated) git-only
|
||||
fallback (:func:`app.rag.git_sources.effective_sources`, the same
|
||||
shared resolver the in-app Sync button uses). Git rows are cloned
|
||||
(first run, shallow ``--depth 1``) or fast-forwarded
|
||||
(first run, full history — no ``--depth``; an existing shallow
|
||||
checkout is unshallowed first, phase 107) or fast-forwarded
|
||||
(``git pull --ff-only``) into ``BOR_SOURCES_DIR/<repo-name>/``
|
||||
(default ``~/bor-sources``); local rows are the existing directories
|
||||
themselves, walked directly. A failing clone/pull — or a local
|
||||
|
||||
Reference in New Issue
Block a user