phase: 107_git_full_history_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 12s

All green — verification complete. Final report:

**Phase 107 — final verification pass (all tasks already in `complete/`)**

- **Verified:** `clone_or_pull` full-history fix (no `--depth` clone; `--is-shallow-repository` probe → one-time `git fetch --unshallow` self-heal → `git pull --ff-only`; D12 fail-loud) in `scripts/git_sync.py`; E2E suite `tests/e2e/test_git_source_dates.py` present; all five stale shallow/tip-date narratives corrected (no residual claims repo-wide; `docs_push.py` `--depth` fetches untouched as designed).
- **Unit+integration:** `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v --no-cov` → **66 passed** (D10 tip-date test replaced by `test_url_clone_yields_true_per_file_dates` + `test_existing_shallow_checkout_self_heals`; unit pins: clone argv w/o `--depth`, probe→unshallow→pull order, unshallow-failure → `GitSyncError`).
- **New E2E in isolation:** `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` → **4 passed** (old file 2020-06-15 ≠ tip file 2024-06-15 in `GET /api/docs`, Sources `Created`/`Updated` columns, viewer `Created` badge ISO title; folder `updated_at` subtree maxes).
- **Phase-106 regressions in isolation:** `test_document_dates.py` → 6 passed; `test_sync_button.py` → 3 passed; `test_git_sources_admin.py` → 6 passed.
- **Full gates:** `uv run pytest` → **2302 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check . && uv run pyright` → **clean, 0 errors**.
- **Completion criteria:** 1 ✅ non-shallow URL clone + true per-file dates · 2 ✅ self-heal / plain-pull / fail-loud · 3 ✅ both suites green, bug-pin replaced · 4 ✅ E2E green in isolation · 5 ✅ all regressions + full suite + coverage + lint · 6 ✅ narratives corrected · 7 — no commit made (harness override: changes left in working tree; task files already in `complete/`).
- **No defects found; no deviations.** Next pending phase: **108_history_wire_check**.
This commit is contained in:
2026-09-13 23:21:39 -04:00
parent addbd4ca08
commit 6bcee831ec
24 changed files with 1337 additions and 74 deletions
+61 -30
View File
@@ -1,8 +1,9 @@
"""Git source sync for import_docs (phase 28).
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
the first time, or fast-forwards an existing checkout with ``git pull
--ff-only`` on subsequent runs.
clone_or_pull(url, dest) clones ``url`` into ``dest`` with FULL history
the first time, or — for an existing checkout — unshallows a shallow
one first (``git fetch --unshallow``, the one-time self-heal) and then
fast-forwards it with ``git pull --ff-only``.
Auth: nothing special — an ``https://…`` URL uses the OS credential
helper / prompts; a ``git@host:repo.git`` URL uses the machine's SSH key.
@@ -11,22 +12,34 @@ used.
This module is the only place the ``git`` CLI is invoked (A11: stdlib
``subprocess`` only, no new packages) — every git command goes through
:func:`run_git`: the clone/pull in :func:`clone_or_pull`, the per-file
last-commit-date walk in :func:`file_commit_dates` (phase 106), and the
docs-push sequence in :mod:`app.core.docs_push` (phase 59).
:func:`run_git`: the clone, the shallow probe, the one-time
``git fetch --unshallow``, and the ``git pull --ff-only`` in
:func:`clone_or_pull`, the per-file last-commit-date walk in
:func:`file_commit_dates` (phase 106), and the docs-push sequence in
:mod:`app.core.docs_push` (phase 59).
Per-file last-commit dates (phase 106, D2/D10) — behavior verified
against scratch repos 2026-09-13:
Per-file last-commit dates (phase 106 D2; phase 107 D11 supersedes
phase 106 D10) — behavior verified against scratch and live repos
2026-09-13 / 2026-09-16:
* a LOCAL-PATH checkout made by :func:`clone_or_pull` keeps FULL
history (``git clone --depth 1 /local/path`` prints "--depth is
ignored in local clones" and does not shallow) →
:func:`file_commit_dates` yields TRUE per-file last-commit dates;
* a URL-TRANSPORT checkout (https/ssh/``file://``) is shallow, and in a
shallow clone git reports the TIP commit as every existing file's
last commit (the shallow boundary is each file's history root) → a
uniform per-repo tip date: no intra-repo distortion, a real
cross-source signal, refreshed on every pull.
* every :func:`clone_or_pull` checkout is FULL history for EVERY
transport (https/ssh/``file://``/local-path): a fresh clone carries
no ``--depth`` (D11 — D10's ``--depth 1`` shallow URL clones are
gone), and an existing shallow checkout (made pre-phase 107 — i.e.
every deployed one, live + dev included) is probed with ``git
rev-parse --is-shallow-repository`` and, while shallow, self-healed
with the ONE-TIME ``git fetch --unshallow`` before the usual ``git
pull --ff-only`` — no re-clone;
* on a full-history checkout :func:`file_commit_dates` yields the
TRUE per-file last-commit date for ALL git sources. The old
shallow-clone behavior was the bug the owner reported 2026-09-16:
in a shallow clone git reports the TIP commit as every existing
file's last commit (the shallow boundary is each file's history
root) → a uniform per-repo tip date (``container_bifrost`` shown as
created on the repo tip, months off); after ``git fetch
--unshallow`` on the live homelab checkout (438 commits visible)
the walk returned the true dates — e.g. ``bifrost.md`` 2026-05-05,
not the 2026-09-07 tip.
"""
from __future__ import annotations
@@ -45,23 +58,40 @@ class GitSyncError(RuntimeError):
def clone_or_pull(url: str, dest: Path | str) -> Path:
"""Clone ``url`` into ``dest`` (shallow, first run) or fast-forward it.
"""Clone ``url`` into ``dest`` (full history, first run) or fast-forward it.
- dest without a ``.git`` (or absent) → ``git clone --depth 1 url dest``
(shallow: the KB is re-imported incrementally anyway).
- dest with a ``.git`` → ``git pull --ff-only`` (refuses to merge
unrelated histories — a broken checkout fails loudly rather than
producing a dirty index).
- dest without a ``.git`` (or absent) → ``git clone url dest`` (full
history for every transport — phase 107 D11: no ``--depth``, so a
URL-transport checkout carries the whole commit log, not just the
tip).
- dest with a ``.git`` → probe ``git rev-parse
--is-shallow-repository``; while shallow, ``git fetch
--unshallow`` (the ONE-TIME self-heal for checkouts made shallow
pre-phase 107 — the next sync of a deployed checkout becomes
full-history without a re-clone), then ``git pull --ff-only``
(refuses to merge unrelated histories — a broken checkout fails
loudly rather than producing a dirty index).
Returns the destination path. Raises :class:`GitSyncError` when git is
missing or a git invocation exits non-zero (with git's stderr in the
message, so the caller can name the failing repo + reason).
message, so the caller can name the failing repo + reason). D12
fail-loud: a failed probe/unshallow/pull propagates exactly like a
clone failure — never a silent fallback to tip dates or mtimes (a
continued shallow checkout would silently re-serve the uniform tip
date, i.e. the bug D11 fixes).
"""
dest = Path(dest)
if not dest.exists() or not (dest / ".git").exists():
dest.parent.mkdir(parents=True, exist_ok=True)
run_git(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
run_git(["git", "clone", url, str(dest)], cwd=dest.parent)
else:
shallow = (
run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest)
.strip()
== "true"
)
if shallow:
run_git(["git", "fetch", "--unshallow"], cwd=dest)
run_git(["git", "pull", "--ff-only"], cwd=dest)
return dest
@@ -104,11 +134,12 @@ def file_commit_dates(dest: Path | str) -> dict[str, datetime]:
source per sync) → ``{repo-relative POSIX path: last-commit
datetime}``, newest-first so the first sighting of a path wins.
The checkout behavior is pinned (verified 2026-09-13 — see the
module docstring): a local-path ``clone_or_pull`` checkout keeps
FULL history → TRUE per-file dates; a URL-transport checkout is
shallow → the repo's TIP-commit date for every working-tree file
(D10: uniform within the repo, real across sources).
Every ``clone_or_pull`` checkout is FULL history (fresh: no
``--depth``; an existing shallow checkout is unshallowed on its
next sync — phase 107 D11 supersedes phase 106 D10's shallow
tip-date behavior) → TRUE per-file last-commit dates for ALL git
sources, local AND URL (verified 2026-09-16 — see the module
docstring).
Fail-soft (pinned): a missing/non-directory checkout, a git failure
(:class:`GitSyncError`), or ANY parse anomaly logs a warning and