"""Git source sync for import_docs (phase 28). clone_or_pull(url, dest, branch=None) clones ``url`` into ``dest`` with FULL history the first time — ``git clone --branch `` when a branch is pinned (phase 135: the source's branch is checked out from the first commit), plain ``git clone`` when not — or, for an existing checkout, brings it to the head of the tracked ref with the UNIFIED ff-or-reset sequence (phase 136 — it replaces both the legacy ``git pull --ff-only`` and phase 135's fetch + ff-merge pair): the phase-135 HEAD guard first when a branch is pinned (a checkout on any other branch fails loudly — the source must be removed and re-added to switch), the tracked ref — the pinned branch, or the checkout's own branch when ``branch=None`` (a detached HEAD fails loudly, D12) — unshallows a shallow one first (``git fetch --unshallow``, the one-time self-heal), ``git fetch origin `` (a fetch FAILURE still fails loudly), and ``git merge --ff-only FETCH_HEAD`` — when the fast-forward is impossible (a force push / any divergent remote head) the checkout is ``git reset --hard FETCH_HEAD`` to the ref's head with ONE loud WARNING line and the function returns normally (a force push is a logged, self-healing event, NEVER a failed sync — the checkout is a cache of the remote head; the legacy ``git pull --ff-only`` is deliberately gone from ``branch=None``: every row gets the ff-or-reset update). Auth: nothing special — an ``https://…`` URL uses the OS credential helper / prompts; a ``git@host:repo.git`` URL uses the machine's SSH key. No credentials are stored here; whatever the URL/SSH config supplies is used. This module is the only place the ``git`` CLI is invoked (A11: stdlib ``subprocess`` only, no new packages) — every git command goes through :func:`run_git`: the clone (with ``--branch`` when a branch is pinned, phase 135), the branch guard's / ref-resolution's ``git rev-parse --abbrev-ref HEAD`` probe, the shallow probe, the one-time ``git fetch --unshallow``, the unified update's ``git fetch origin `` + ``git merge --ff-only FETCH_HEAD`` pair, and the phase-136 force-push fallback's ``git rev-parse HEAD`` / ``git rev-parse FETCH_HEAD`` capture + ``git reset --hard FETCH_HEAD`` in :func:`clone_or_pull`; the per-file last-commit-date walk in :func:`file_commit_dates` (phase 106 — it walks the checkout's own log, so a branch checkout yields the BRANCH's history: the head-of-branch semantics phase 135 pins, untouched — a post-reset checkout yields the NEW head's dates, the same semantics); and the docs-push sequence in :mod:`app.core.docs_push` (phase 59). Per-file last-commit dates (phase 106 D2; phase 107 D11 supersedes phase 106 D10) — behavior verified against scratch and live repos 2026-09-13 / 2026-09-16: * every :func:`clone_or_pull` checkout is FULL history for EVERY transport (https/ssh/``file://``/local-path): a fresh clone carries no ``--depth`` (D11 — D10's ``--depth 1`` shallow URL clones are gone; ``--branch`` clones, phase 135, included), and an existing shallow checkout (made pre-phase 107 — i.e. every deployed one, live + dev included) is probed with ``git rev-parse --is-shallow-repository`` and, while shallow, self-healed with the ONE-TIME ``git fetch --unshallow`` before the unified update's ``git fetch origin `` + ``git merge --ff-only FETCH_HEAD`` (+ the phase-136 ``git reset --hard FETCH_HEAD`` fallback on a non-ff merge) — no re-clone; * D11/D12, extended by phase 136 (the force-push contract): a non-ff merge resets to the ref's head instead of failing — the checkout is a cache of the remote head, and "always the head of the branch" is the invariant ``clone_or_pull`` enforces. A force push (or any divergent remote head) is a LOGGED, SELF-HEALING event — one WARNING line, then the run continues — never a failed sync; everything else still fails loud per D12 (a fetch failure, a broken probe/unshallow, a detached HEAD, a branch mismatch); * on a full-history checkout :func:`file_commit_dates` yields the TRUE per-file last-commit date for ALL git sources. The old shallow-clone behavior was the bug the owner reported 2026-09-16: in a shallow clone git reports the TIP commit as every existing file's last commit (the shallow boundary is each file's history root) → a uniform per-repo tip date (``container_bifrost`` shown as created on the repo tip, months off); after ``git fetch --unshallow`` on the live homelab checkout (438 commits visible) the walk returned the true dates — e.g. ``bifrost.md`` 2026-05-05, not the 2026-09-07 tip. """ from __future__ import annotations import logging import subprocess from datetime import datetime from pathlib import Path logger = logging.getLogger(__name__) __all__ = ["GitSyncError", "clone_or_pull", "file_commit_dates", "run_git"] class GitSyncError(RuntimeError): """A git clone/pull failed (or git is missing); carries git's stderr.""" def clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path: """Clone ``url`` into ``dest`` (full history, first run) or bring the existing checkout to the tracked ref's head (ff-or-reset, phase 136). - dest without a ``.git`` (or absent) → ``git clone url dest`` (full history for every transport — phase 107 D11: no ``--depth``, so a URL-transport checkout carries the whole commit log, not just the tip); with a pinned ``branch`` → ``git clone --branch url dest`` (phase 135: the source's branch is checked out from the first commit — no post-clone checkout; ``branch=None`` keeps the two-argument argv byte-identical). - dest with a ``.git`` (ANY ``branch``) → the UNIFIED update sequence (phase 136 — it replaces both the legacy ``git pull --ff-only`` and phase 135's fetch + ff-merge pair), in order: 1. the tracked REF: a pinned ``branch`` keeps the HEAD GUARD (phase 135, UNCHANGED and first — ``git rev-parse --abbrev-ref HEAD``; a checkout sitting on any other branch raises the remove-and-re-add :class:`GitSyncError`, naming the checkout and BOTH branch names — never the URL, so no credential exposure — BEFORE any update; no silent branch switch, since the checkout may serve other rows' documents) and ``ref = branch``; ``branch=None`` resolves the checkout's OWN branch (the pre-phase default-branch behavior) — and a DETACHED checkout (the result ``"HEAD"``) raises the remove-and-re-add :class:`GitSyncError` naming the checkout (D12 fail-loud: the app never CREATES a detached checkout — a fresh clone always lands on a branch — so this is a genuinely broken checkout, not a force-push case). 2. the shallow probe + one-time ``git fetch --unshallow`` self-heal (UNCHANGED — phase 107 D11). 3. ``git fetch origin `` — a fetch FAILURE (network/auth, the branch gone from the remote, git missing) STILL raises :class:`GitSyncError` (a loud failure — a force push is NOT a fetch failure; the reset is scoped to the merge step only). 4. try ``git merge --ff-only FETCH_HEAD``. 5. ON MERGE FAILURE — the reset fallback (phase 136): capture ``git rev-parse HEAD`` (old) and ``git rev-parse FETCH_HEAD`` (new), run ``git reset --hard FETCH_HEAD``, log ONE WARNING line (PLAN §9: ``git sync: divergent remote head on (force push) — reset → to origin/`` — the checkout PATH, never the URL, credential-safety) carrying the merge's stderr for diagnosability, and RETURN the dest normally (the sync proceeds — a force push is a logged, self-healing event, NEVER a failed sync; the merge's stderr rides the log only — there is no user-facing error). The checkout is a CACHE of the remote head (the app owns it; the KB is rebuilt from it on the same run), so "always the head of the branch" is the invariant this sequence enforces. Returns the destination path. Raises :class:`GitSyncError` when git is missing or a git invocation exits non-zero (with git's stderr in the message, so the caller can name the failing repo + reason) — EXCEPT the non-ff merge, which takes the reset fallback of step 5 instead of failing (phase 136). D12 fail-loud still governs everything the reset does not cover: a failed probe/unshallow/fetch propagates exactly like a clone failure — never a silent fallback to tip dates or mtimes (a continued shallow checkout would silently re-serve the uniform tip date, i.e. the bug D11 fixes) and never a silent branch switch (phase 135 — the fail-loud guard, step 1). """ dest = Path(dest) if not dest.exists() or not (dest / ".git").exists(): dest.parent.mkdir(parents=True, exist_ok=True) argv = ["git", "clone"] if branch is not None: argv += ["--branch", branch] argv += [url, str(dest)] run_git(argv, cwd=dest.parent) else: # existing checkout (the fresh-clone path above is untouched): if branch is not None: # Phase 135 HEAD guard, UNCHANGED (runs first, before any # update): a mismatched checkout fails loudly. The message # names the checkout and both branch names; it never # carries the URL (the only place a token could ride). current = run_git(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=dest).strip() if current != branch: raise GitSyncError( f"checkout {dest} is on branch '{current}' but the source " f"specifies branch '{branch}' — remove and re-add the " "source to switch branches" ) ref = branch else: # branch=None: the tracked ref is the checkout's OWN branch # (the pre-phase default-branch behavior). A detached HEAD # (the result "HEAD") is a genuinely broken checkout — the # app never creates one — and fails loud (D12), not a # force-push case. ref = run_git(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=dest).strip() if ref == "HEAD": # detached — the app never creates one (D12 fail-loud) raise GitSyncError( f"checkout {dest} is in a detached-HEAD state — remove and " "re-add the source to rebuild it" ) _unshallow_if_shallow(dest) # A fetch FAILURE (network/auth, the branch gone from the # remote, git missing) STILL raises — a force push is not a # fetch failure; the reset below is scoped to the merge step. run_git(["git", "fetch", "origin", ref], cwd=dest) try: run_git(["git", "merge", "--ff-only", "FETCH_HEAD"], cwd=dest) except GitSyncError as ff: # A divergent remote head (a force push, or any non-ff # state): the checkout is a cache of the remote head — # reset it to the tracked ref's head and continue. The ONE # WARNING line (PLAN §9) is the only observable side effect # besides the checkout state: no user-facing error, no # status change — the run continues. The merge's stderr # rides the log (diagnosability) but never a user-facing # error (there is no error). The log carries the checkout # PATH, never the URL (credential-safety). old = run_git(["git", "rev-parse", "HEAD"], cwd=dest).strip() new = run_git(["git", "rev-parse", "FETCH_HEAD"], cwd=dest).strip() run_git(["git", "reset", "--hard", "FETCH_HEAD"], cwd=dest) logger.warning( "git sync: divergent remote head on %s (force push) — reset %s → %s to " "origin/%s (%s)", dest, old[:7], new[:7], ref, ff, ) return dest def _unshallow_if_shallow(dest: Path) -> None: """The one-time shallow self-heal (phase 107 D11). Probe ``git rev-parse --is-shallow-repository``; while shallow, ``git fetch --unshallow`` (a pre-phase-107 deployed checkout becomes full-history on its next sync without a re-clone). The UNIFIED existing-checkout update of :func:`clone_or_pull` (phase 136 — one sequence for both ``branch=None`` and a pinned branch) runs this before its fetch (the unit argv pins are the proof). """ shallow = ( run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest) .strip() == "true" ) if shallow: run_git(["git", "fetch", "--unshallow"], cwd=dest) def _parse_commit_dates(output: str) -> dict[str, datetime]: """Parse ``git log --name-only --format=@@%cI`` output (newest first). A ``@@`` line starts a commit (``%cI`` is ISO-strict, so the date is always aware — parsed with ``datetime.fromisoformat``); the following non-empty, non-``@@`` lines are repo-relative paths. The FIRST sighting of a path wins (the walk is newest-first) — that is the file's last-commit date. Paths are split on whitespace (like name-only output), ``\\``-normalized to ``/``, and a leading ``/`` is stripped. Raises ``ValueError`` on a malformed commit date or a path line before any commit header (the caller fails soft). """ dates: dict[str, datetime] = {} commit: datetime | None = None for line in output.splitlines(): line = line.strip() if not line: continue if line.startswith("@@"): commit = datetime.fromisoformat(line[2:]) continue if commit is None: raise ValueError(f"path line before any commit header: {line!r}") for raw_path in line.split(): path = raw_path.replace("\\", "/").lstrip("/") if path: dates.setdefault(path, commit) return dates def file_commit_dates(dest: Path | str) -> dict[str, datetime]: """Per-file last-commit dates for one checkout (phase 106, D2). ONE ``git log --name-only --format=@@%cI`` walk through :func:`run_git` (the A11 single-invocation site, one git call per source per sync) → ``{repo-relative POSIX path: last-commit datetime}``, newest-first so the first sighting of a path wins. Every ``clone_or_pull`` checkout is FULL history (fresh: no ``--depth``; an existing shallow checkout is unshallowed on its next sync — phase 107 D11 supersedes phase 106 D10's shallow tip-date behavior) → TRUE per-file last-commit dates for ALL git sources, local AND URL (verified 2026-09-16 — see the module docstring). Fail-soft (pinned): a missing/non-directory checkout, a git failure (:class:`GitSyncError`), or ANY parse anomaly logs a warning and returns ``{}`` — the importer falls back to file mtimes; a date walk must never break a sync. """ dest = Path(dest) if not dest.is_dir(): logger.warning( "file_commit_dates: %s is not a directory — no git dates " "(the importer will fall back to file mtimes)", dest, ) return {} try: output = run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest) except GitSyncError as exc: logger.warning( "file_commit_dates: git log failed for %s: %s — the importer " "will fall back to file mtimes", dest, exc, ) return {} try: return _parse_commit_dates(output) except ValueError as exc: logger.warning( "file_commit_dates: unparseable git log output for %s (%s) — the " "importer will fall back to file mtimes", dest, exc, ) return {} def run_git(argv: list[str], cwd: Path) -> str: """Run one git command, capturing output; raise GitSyncError on failure. The single ``git`` invocation point for the whole app (A11). Every step of :func:`clone_or_pull` and of the docs-push sequence (:mod:`app.core.docs_push`, phase 59) goes through here, so error handling stays uniform: captured stdout on success, and :class:`GitSyncError` carrying git's stderr on a non-zero exit (or when the git binary is missing from PATH). """ try: proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True) except FileNotFoundError: raise GitSyncError("git was not found on PATH — install git and retry") from None if proc.returncode != 0: raise GitSyncError( f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): " f"{proc.stderr.strip()}" ) return proc.stdout