brain-of-reese/scripts/git_sync.py

"""Git source sync for import_docs (phase 28).

clone_or_pull(url, dest, branch=None) clones ``url`` into ``dest`` with
FULL history the first time — ``git clone --branch <b>`` when a branch
is pinned (phase 135: the source's branch is checked out from the
first commit), plain ``git clone`` when not — or, for an existing
checkout, brings it to the head of the tracked ref with the UNIFIED
ff-or-reset sequence (phase 136 — it replaces both the legacy
``git pull --ff-only`` and phase 135's fetch + ff-merge pair): the
phase-135 HEAD guard first when a branch is pinned (a checkout on any
other branch fails loudly — the source must be removed and re-added
to switch), the tracked ref — the pinned branch, or the checkout's
own branch when ``branch=None`` (a detached HEAD fails loudly, D12) —
unshallows a shallow one first (``git fetch --unshallow``, the
one-time self-heal), ``git fetch origin <ref>`` (a fetch FAILURE
still fails loudly), and ``git merge --ff-only FETCH_HEAD`` — when
the fast-forward is impossible (a force push / any divergent remote
head) the checkout is ``git reset --hard FETCH_HEAD`` to the ref's
head with ONE loud WARNING line and the function returns normally
(a force push is a logged, self-healing event, NEVER a failed sync —
the checkout is a cache of the remote head; the legacy ``git pull
--ff-only`` is deliberately gone from ``branch=None``: every row
gets the ff-or-reset update).

Auth: nothing special — an ``https://…`` URL uses the OS credential
helper / prompts; a ``git@host:repo.git`` URL uses the machine's SSH key.
No credentials are stored here; whatever the URL/SSH config supplies is
used.

This module is the only place the ``git`` CLI is invoked (A11: stdlib
``subprocess`` only, no new packages) — and since phase 186 there are
exactly TWO spawn sites here, one per shape, both owning the same
``argv`` → :class:`GitSyncError` contract:

* :func:`run_git` — the BUFFERED site (``subprocess.run``, the whole
  output returned as one string): the clone (with ``--branch`` when a
branch is pinned, phase 135), the branch guard's / ref-resolution's
``git rev-parse --abbrev-ref HEAD`` probe, the shallow probe, the
one-time ``git fetch --unshallow``, the unified update's ``git fetch
origin <ref>`` + ``git merge --ff-only FETCH_HEAD`` pair, and the
phase-136 force-push fallback's ``git rev-parse HEAD`` / ``git
rev-parse FETCH_HEAD`` capture + ``git reset --hard FETCH_HEAD`` in
:func:`clone_or_pull`, and the docs-push sequence in
:mod:`app.core.docs_push` (phase 59) — every caller but the commit-date
walk, which moved to the streaming sibling below in phase 186 (A4:
``run_git`` keeps its shape; migrating the others is churn without a
measured cost).
* :func:`stream_git_lines` — the STREAMING site added by phase 186
  (``subprocess.Popen`` + line-at-a-time iteration) — used ONLY by the
  commit-date walk, because a full-history ``git log`` scales with a
  repository's ENTIRE history, not its working tree, and holding it as
  one string (plus the parser's ``splitlines()`` copy) cost tens of MB
  of transient peak per source on the 2026-10-15 incident box.

Both sites are reached from async code ONLY through
:func:`asyncio.to_thread` (phase 179 / ``AGENTS.md`` rule 11) — neither
may ever run on the event loop. The ONE enumerated spawn outside these
two functions in the whole app is the boot-time asset-version probe in
:mod:`app.core.caching` (a 5-second-timeouted ``git rev-parse --short
HEAD`` that never raises and is not part of any sync path); the unit
pin ``test_only_two_git_spawn_sites`` enumerates it, so a NEW spawn
site anywhere in ``app/`` or ``scripts/`` fails the suite.

Per-file last-commit dates (phase 106 D2; phase 107 D11 supersedes
phase 106 D10) — behavior verified against scratch and live repos
2026-09-13 / 2026-09-16:

* every :func:`clone_or_pull` checkout is FULL history for EVERY
  transport (https/ssh/``file://``/local-path): a fresh clone carries
  no ``--depth`` (D11 — D10's ``--depth 1`` shallow URL clones are
gone; ``--branch`` clones, phase 135, included), and an existing
  shallow checkout (made pre-phase 107 — i.e. every deployed one,
  live + dev included) is probed with ``git
  rev-parse --is-shallow-repository`` and, while shallow, self-healed
  with the ONE-TIME ``git fetch --unshallow`` before the unified
  update's ``git fetch origin <ref>`` + ``git merge --ff-only
  FETCH_HEAD`` (+ the phase-136 ``git reset --hard FETCH_HEAD``
  fallback on a non-ff merge) — no re-clone;
* D11/D12, extended by phase 136 (the force-push contract): a
  non-ff merge resets to the ref's head instead of failing — the
  checkout is a cache of the remote head, and "always the head of
  the branch" is the invariant ``clone_or_pull`` enforces. A force
  push (or any divergent remote head) is a LOGGED, SELF-HEALING
  event — one WARNING line, then the run continues — never a failed
  sync; everything else still fails loud per D12 (a fetch failure,
  a broken probe/unshallow, a detached HEAD, a branch mismatch);
* on a full-history checkout :func:`file_commit_dates` yields the
  TRUE per-file last-commit date for ALL git sources. The old
  shallow-clone behavior was the bug the owner reported 2026-09-16:
  in a shallow clone git reports the TIP commit as every existing
  file's last commit (the shallow boundary is each file's history
  root) → a uniform per-repo tip date (``container_bifrost`` shown as
  created on the repo tip, months off); after ``git fetch
  --unshallow`` on the live homelab checkout (438 commits visible)
  the walk returned the true dates — e.g. ``bifrost.md`` 2026-05-05,
  not the 2026-09-07 tip.
"""
from __future__ import annotations

import contextlib
import logging
import subprocess
import threading
from collections.abc import Iterable, Iterator
from datetime import datetime
from pathlib import Path

logger = logging.getLogger(__name__)

__all__ = [
    "GitSyncError",
    "clone_or_pull",
    "file_commit_dates",
    "run_git",
    "stream_git_lines",
]


class GitSyncError(RuntimeError):
    """A git clone/pull failed (or git is missing); carries git's stderr."""


def clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path:
    """Clone ``url`` into ``dest`` (full history, first run) or bring the
    existing checkout to the tracked ref's head (ff-or-reset, phase 136).

    - dest without a ``.git`` (or absent) → ``git clone url dest`` (full
      history for every transport — phase 107 D11: no ``--depth``, so a
      URL-transport checkout carries the whole commit log, not just the
      tip); with a pinned ``branch`` → ``git clone --branch <b> url
      dest`` (phase 135: the source's branch is checked out from the
      first commit — no post-clone checkout; ``branch=None`` keeps the
      two-argument argv byte-identical).
    - dest with a ``.git`` (ANY ``branch``) → the UNIFIED update
      sequence (phase 136 — it replaces both the legacy
      ``git pull --ff-only`` and phase 135's fetch + ff-merge pair), in
      order:
      1. the tracked REF: a pinned ``branch`` keeps the HEAD GUARD
         (phase 135, UNCHANGED and first — ``git rev-parse
         --abbrev-ref HEAD``; a checkout sitting on any other branch
         raises the remove-and-re-add :class:`GitSyncError`, naming
         the checkout and BOTH branch names — never the URL, so no
         credential exposure — BEFORE any update; no silent branch
         switch, since the checkout may serve other rows' documents)
         and ``ref = branch``; ``branch=None`` resolves the checkout's
         OWN branch (the pre-phase default-branch behavior) — and a
         DETACHED checkout (the result ``"HEAD"``) raises the
         remove-and-re-add :class:`GitSyncError` naming the checkout
         (D12 fail-loud: the app never CREATES a detached checkout —
         a fresh clone always lands on a branch — so this is a
         genuinely broken checkout, not a force-push case).
      2. the shallow probe + one-time ``git fetch --unshallow``
         self-heal (UNCHANGED — phase 107 D11).
      3. ``git fetch origin <ref>`` — a fetch FAILURE (network/auth,
         the branch gone from the remote, git missing) STILL raises
         :class:`GitSyncError` (a loud failure — a force push is NOT a
         fetch failure; the reset is scoped to the merge step only).
      4. try ``git merge --ff-only FETCH_HEAD``.
      5. ON MERGE FAILURE — the reset fallback (phase 136): capture
         ``git rev-parse HEAD`` (old) and ``git rev-parse FETCH_HEAD``
         (new), run ``git reset --hard FETCH_HEAD``, log ONE WARNING
         line (PLAN §9: ``git sync: divergent remote head on <dest>
         (force push) — reset <old[:7]> → <new[:7]> to
         origin/<ref>`` — the checkout PATH, never the URL,
         credential-safety) carrying the merge's stderr for
         diagnosability, and RETURN the dest normally (the sync
         proceeds — a force push is a logged, self-healing event,
         NEVER a failed sync; the merge's stderr rides the log only —
         there is no user-facing error). The checkout is a CACHE of
         the remote head (the app owns it; the KB is rebuilt from it
         on the same run), so "always the head of the branch" is the
         invariant this sequence enforces.

    Returns the destination path. Raises :class:`GitSyncError` when git is
    missing or a git invocation exits non-zero (with git's stderr in the
    message, so the caller can name the failing repo + reason) — EXCEPT
    the non-ff merge, which takes the reset fallback of step 5 instead
    of failing (phase 136). D12 fail-loud still governs everything the
    reset does not cover: a failed probe/unshallow/fetch propagates
    exactly like a clone failure — never a silent fallback to tip
    dates or mtimes (a continued shallow checkout would silently
    re-serve the uniform tip date, i.e. the bug D11 fixes) and never a
    silent branch switch (phase 135 — the fail-loud guard, step 1).
    """
    dest = Path(dest)
    if not dest.exists() or not (dest / ".git").exists():
        dest.parent.mkdir(parents=True, exist_ok=True)
        argv = ["git", "clone"]
        if branch is not None:
            argv += ["--branch", branch]
        argv += [url, str(dest)]
        run_git(argv, cwd=dest.parent)
    else:
        # existing checkout (the fresh-clone path above is untouched):
        if branch is not None:
            # Phase 135 HEAD guard, UNCHANGED (runs first, before any
            # update): a mismatched checkout fails loudly. The message
            # names the checkout and both branch names; it never
            # carries the URL (the only place a token could ride).
            current = run_git(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=dest).strip()
            if current != branch:
                raise GitSyncError(
                    f"checkout {dest} is on branch '{current}' but the source "
                    f"specifies branch '{branch}' — remove and re-add the "
                    "source to switch branches"
                )
            ref = branch
        else:
            # branch=None: the tracked ref is the checkout's OWN branch
            # (the pre-phase default-branch behavior). A detached HEAD
            # (the result "HEAD") is a genuinely broken checkout — the
            # app never creates one — and fails loud (D12), not a
            # force-push case.
            ref = run_git(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=dest).strip()
            if ref == "HEAD":  # detached — the app never creates one (D12 fail-loud)
                raise GitSyncError(
                    f"checkout {dest} is in a detached-HEAD state — remove and "
                    "re-add the source to rebuild it"
                )
        _unshallow_if_shallow(dest)
        # A fetch FAILURE (network/auth, the branch gone from the
        # remote, git missing) STILL raises — a force push is not a
        # fetch failure; the reset below is scoped to the merge step.
        run_git(["git", "fetch", "origin", ref], cwd=dest)
        try:
            run_git(["git", "merge", "--ff-only", "FETCH_HEAD"], cwd=dest)
        except GitSyncError as ff:
            # A divergent remote head (a force push, or any non-ff
            # state): the checkout is a cache of the remote head —
            # reset it to the tracked ref's head and continue. The ONE
            # WARNING line (PLAN §9) is the only observable side effect
            # besides the checkout state: no user-facing error, no
            # status change — the run continues. The merge's stderr
            # rides the log (diagnosability) but never a user-facing
            # error (there is no error). The log carries the checkout
            # PATH, never the URL (credential-safety).
            old = run_git(["git", "rev-parse", "HEAD"], cwd=dest).strip()
            new = run_git(["git", "rev-parse", "FETCH_HEAD"], cwd=dest).strip()
            run_git(["git", "reset", "--hard", "FETCH_HEAD"], cwd=dest)
            logger.warning(
                "git sync: divergent remote head on %s (force push) — reset %s → %s to "
                "origin/%s (%s)",
                dest, old[:7], new[:7], ref, ff,
            )
    return dest


def _unshallow_if_shallow(dest: Path) -> None:
    """The one-time shallow self-heal (phase 107 D11).

    Probe ``git rev-parse --is-shallow-repository``; while shallow,
    ``git fetch --unshallow`` (a pre-phase-107 deployed checkout
    becomes full-history on its next sync without a re-clone). The
    UNIFIED existing-checkout update of :func:`clone_or_pull` (phase
    136 — one sequence for both ``branch=None`` and a pinned branch)
    runs this before its fetch (the unit argv pins are the proof).
    """
    shallow = (
        run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest)
        .strip()
        == "true"
    )
    if shallow:
        run_git(["git", "fetch", "--unshallow"], cwd=dest)


def _parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]:
    """Parse ``git log --name-only --format=@@%cI`` output (newest first).

    *lines* may be a string or any iterable of lines (phase 186 —
    :func:`stream_git_lines` feeds it a live pipe); a string input
    behaves exactly as before.

    A ``@@<ISO-8601>`` line starts a commit (``%cI`` is ISO-strict, so
    the date is always aware — parsed with ``datetime.fromisoformat``);
    the following non-empty, non-``@@`` lines are repo-relative paths.
    The FIRST sighting of a path wins (the walk is newest-first) — that
    is the file's last-commit date. Paths are split on whitespace (like
    name-only output), ``\\``-normalized to ``/``, and a leading ``/``
    is stripped. Raises ``ValueError`` on a malformed commit date or a
    path line before any commit header (the caller fails soft).
    """
    dates: dict[str, datetime] = {}
    commit: datetime | None = None
    if isinstance(lines, str):
        lines = lines.splitlines()
    for line in lines:
        line = line.strip()
        if not line:
            continue
        if line.startswith("@@"):
            commit = datetime.fromisoformat(line[2:])
            continue
        if commit is None:
            raise ValueError(f"path line before any commit header: {line!r}")
        for raw_path in line.split():
            path = raw_path.replace("\\", "/").lstrip("/")
            if path:
                dates.setdefault(path, commit)
    return dates


def file_commit_dates(dest: Path | str) -> dict[str, datetime]:
    """Per-file last-commit dates for one checkout (phase 106, D2).

    ONE ``git log --name-only --format=@@%cI`` walk through
    :func:`stream_git_lines` (one git call per source per sync — since
    phase 186 the STREAMING of the two A11 sites, parsed straight off
    the pipe; :func:`run_git` is its buffered sibling) →
    ``{repo-relative POSIX path: last-commit datetime}``, newest-first
    so the first sighting of a path wins.

    Every ``clone_or_pull`` checkout is FULL history (fresh: no
    ``--depth``; an existing shallow checkout is unshallowed on its
    next sync — phase 107 D11 supersedes phase 106 D10's shallow
    tip-date behavior) → TRUE per-file last-commit dates for ALL git
    sources, local AND URL (verified 2026-09-16 — see the module
    docstring).

    Fail-soft (pinned): a missing/non-directory checkout, a git failure
    (:class:`GitSyncError`), or ANY parse anomaly logs a warning and
    returns ``{}`` — the importer falls back to file mtimes; a date
    walk must never break a sync.

    Phase 186 — the walk STREAMS (the owner incident of 2026-10-15).
    The output arrives line by line from :func:`stream_git_lines`
    instead of as one ``run_git`` string, so a full-history ``git log``
    is never held in RAM (the incident's transient peak: 346,829 lines
    / 15 MB buffered TWICE — the string plus the parser's
    ``splitlines()`` list — for one source). Two consequences stated,
    not hidden: (1) a git failure that happens MID-stream now surfaces
    AFTER some lines were parsed, and the method still returns ``{}``
    with the same warning — the phase-106 D2 fail-soft contract is
    about the RESULT (``{}`` + the mtime fallback), and the result is
    unchanged; (2) the dict that IS returned — one entry per path in
    the repository's history — stays retained by ``_run_sync`` in
    ``doc_dates_by_root`` for the whole run (phase 186 A3, OUT OF
    SCOPE: bounding it changes document DATES, not just memory).
    """
    dest = Path(dest)
    if not dest.is_dir():
        logger.warning(
            "file_commit_dates: %s is not a directory — no git dates "
            "(the importer will fall back to file mtimes)",
            dest,
        )
        return {}
    try:
        # Phase 186: the streaming site — no whole ``git log`` in RAM.
        return _parse_commit_dates(
            stream_git_lines(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)
        )
    except GitSyncError as exc:
        logger.warning(
            "file_commit_dates: git log failed for %s: %s — the importer "
            "will fall back to file mtimes",
            dest,
            exc,
        )
        return {}
    except ValueError as exc:
        logger.warning(
            "file_commit_dates: unparseable git log output for %s (%s) — the "
            "importer will fall back to file mtimes",
            dest,
            exc,
        )
        return {}


def run_git(
    argv: list[str],
    cwd: Path,
    env: dict[str, str] | None = None,
) -> str:
    """Run one git command, capturing output; raise GitSyncError on failure.

    The single ``git`` invocation point for the whole app (A11). Every
    step of :func:`clone_or_pull` and of the docs-push sequence
    (:mod:`app.core.docs_push`, phase 59) goes through here, so error
    handling stays uniform: captured stdout on success, and
    :class:`GitSyncError` carrying git's stderr on a non-zero exit (or
    when the git binary is missing from PATH).

    ``env`` is passed through verbatim to :func:`subprocess.run`:
    ``None`` (the default — the sync path's only form) inherits the
    parent environment. **The subprocess env-replacement trap:** a dict
    REPLACES the parent environment, so a caller that needs add-on
    variables (the docs-push credential env, C2) must build a complete
    environment (a merge over ``os.environ``) itself.
    """
    try:
        proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True, env=env)
    except FileNotFoundError:
        raise GitSyncError("git was not found on PATH — install git and retry") from None
    if proc.returncode != 0:
        raise GitSyncError(
            f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): "
            f"{proc.stderr.strip()}"
        )
    return proc.stdout


def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
    """Stream one git command's stdout, line by line (phase 186, A11 sibling).

    The streaming counterpart of :func:`run_git` and the SECOND of the
    module's exactly-two ``subprocess`` spawn sites (the unit pin
    :func:`test_only_two_git_spawn_sites` enforces it): the commit-date
    walk uses it so a repository's whole history never exists as one
    ``str`` — each line is yielded straight off the pipe and can be
    dropped afterwards.

    Shape (locked A2 — the deadlock-free form):

    * ``Popen`` with BOTH ``stdout`` and ``stderr`` piped and
      ``text=True``; ``env`` stays ``None``, i.e. the parent environment
      is inherited — the same semantics as :func:`run_git` (the
      subprocess env-replacement trap documented there does not apply
      here because no env dict is ever built);
    * ``stderr`` is drained by a DAEMON THREAD from before stdout is
      consumed. A second pipe nobody reads fills up (~64 KB) and
      blocks the child while the parent blocks reading ``stdout`` — the
      classic two-pipe deadlock. ``stderr=STDOUT`` would corrupt the
      parse, ``stderr=DEVNULL`` would lose the text
      :class:`GitSyncError` carries, and ``communicate()`` after the
      fact is impossible once stdout has been consumed: the thread is
      the only correct option;
    * on a non-zero exit, :class:`GitSyncError` with the message shape
      copied from :func:`run_git` verbatim (``git <argv[1:]> failed
      (exit N): <stderr>``) so every caller's fail-soft handling and
      log line is byte-identical to the buffered path;
    * ``finally``-safe: a consumer that abandons the generator (early
      ``break``, an exception in the parser, ``GeneratorExit`` from
      ``close()``) terminates the child instead of leaking it, then
      closes both pipes, joins the drain thread, and reaps.

    Because the error surfaces while the generator is being consumed,
    a mid-stream failure raises AFTER some lines were yielded — the
    only caller (:func:`file_commit_dates`) turns that into its usual
    ``{}`` + mtime fallback, so the observable result is unchanged.
    """
    try:
        proc = subprocess.Popen(
            argv,
            cwd=cwd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            env=None,
        )
    except FileNotFoundError:
        # The same contract as run_git: a missing git binary is a
        # GitSyncError, so file_commit_dates keeps failing soft to mtimes.
        raise GitSyncError("git was not found on PATH — install git and retry") from None
    stderr_lines: list[str] = []

    def _drain_stderr() -> None:
        """Keep git's stderr pipe empty for the whole life of the child."""
        if proc.stderr is not None:
            stderr_lines.extend(proc.stderr)

    err_thread = threading.Thread(target=_drain_stderr, daemon=True, name="git-stderr-drain")
    err_thread.start()
    try:
        # `yield from` keeps the delegate closed on any exit path (the
        # GeneratorExit machinery closes it), and the `or ()` guards the
        # Optional type of ``Popen.stdout`` for the type checker.
        yield from proc.stdout or ()
    except BaseException:
        # The consumer walked away (GeneratorExit on close(), a parse
        # error, a cancelled worker): never leave a `git` child running.
        # An already-exited child just makes terminate() a no-op.
        with contextlib.suppress(OSError):
            proc.terminate()
        raise
    finally:
        if proc.stdout is not None:
            proc.stdout.close()
        err_thread.join()
        if proc.stderr is not None:
            proc.stderr.close()
        proc.wait()
    if proc.returncode != 0:
        raise GitSyncError(
            f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): "
            f"{''.join(stderr_lines).strip()}"
        )