phase: 186_git_log_streaming_dates
## Final report — phase 186 `git_log_streaming_dates`
**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.
### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |
Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.
### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.
Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.
⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).
Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
This commit is contained in:
+547
-3
@@ -5,8 +5,8 @@ matrix).
|
||||
|
||||
``clone_or_pull`` dispatches to ``git clone`` (full history, fresh dest)
|
||||
or the UNIFIED existing-checkout update (phase 136 — one sequence for
|
||||
both branch shapes) with ``subprocess`` fully mocked — the real git
|
||||
CLI is never invoked, so the tests run anywhere: ref resolution
|
||||
both branch shapes) with ``subprocess`` fully mocked — no real git CLI
|
||||
is needed, so the tests run anywhere: ref resolution
|
||||
(``rev-parse --abbrev-ref HEAD`` — the phase-135 HEAD guard for a
|
||||
pinned ``branch``, the checkout's own branch + the detached-HEAD
|
||||
fail-loud for ``branch=None``) → the shallow probe → optional
|
||||
@@ -18,17 +18,54 @@ fail-loud for ``branch=None``) → the shallow probe → optional
|
||||
force push never fails a sync — phase 136's fix). The fresh-clone
|
||||
argv (with/without ``--branch``) is the pre-phase one — the argv pins
|
||||
below are the regression gate for every existing deployment.
|
||||
|
||||
Phase 186 adds the commit-date PARSER pins and the STREAMING git site.
|
||||
``_parse_commit_dates`` accepts ``str | Iterable[str]`` so the walk can
|
||||
stream off the subprocess pipe instead of buffering the whole ``git
|
||||
log`` in RAM, and ``stream_git_lines`` is the A11 sibling of
|
||||
``run_git`` — its success / error-shape / huge-stderr cases run the
|
||||
REAL git CLI through the house ``_git`` helper (skipped, not failed,
|
||||
without it, like ``tests/integration/test_git_file_dates.py``), while
|
||||
the abandon-the-stream case uses a recorded ``Popen`` stand-in so the
|
||||
terminate-and-reap arm is deterministic instead of a race. The
|
||||
``test_only_two_git_spawn_sites`` pin is the AST version of the old A11
|
||||
grep: exactly one ``subprocess`` spawn per shape, both inside
|
||||
``git_sync.py``. The
|
||||
memory pin's synthetic corpus is ~300,000 lines because that is the
|
||||
size of the real thing on the incident box (2026-10-15, measured
|
||||
read-only inside the app container): ``litellm`` 54,502 commits /
|
||||
346,829 lines / 15,077,351 bytes; ``llama.cpp`` 11,659 commits /
|
||||
69,922 lines / 1,867,046 bytes; ``gufo`` 9,155 lines;
|
||||
``vllm-radiance`` 4,855 lines — i.e. the walk scales with a
|
||||
repository's ENTIRE history, not its working tree, which is why the
|
||||
corpus sits at 306,000 lines (3,000 commits × 100 path lines + the
|
||||
header/blank pair per commit) rather than at a toy size.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
import tracemalloc
|
||||
from collections.abc import Callable, Generator, Iterable, Iterator
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
import scripts.git_sync as git_sync
|
||||
from scripts.git_sync import GitSyncError, clone_or_pull
|
||||
from app.core import docs_push
|
||||
from scripts.git_sync import (
|
||||
GitSyncError,
|
||||
_parse_commit_dates, # pyright: ignore[reportPrivateUsage]
|
||||
clone_or_pull,
|
||||
stream_git_lines,
|
||||
)
|
||||
|
||||
|
||||
class _FakeProc:
|
||||
@@ -697,3 +734,510 @@ def test_null_branch_detached_head_fails_loud(
|
||||
assert [call["argv"] for call in calls] == [
|
||||
["git", "rev-parse", "--abbrev-ref", "HEAD"]
|
||||
] # the guard is first: exactly ONE argv ran (no fetch, no reset)
|
||||
|
||||
|
||||
# --- phase 186: the parser takes a stream (semantics untouched) ------------
|
||||
|
||||
#: One commit header of the synthetic corpus, in the exact byte shape
|
||||
#: ``git log --name-only --format=@@%cI`` emits (``%cI`` is ISO-strict).
|
||||
_CORPUS_HEADER = "@@2026-01-01T00:00:00+00:00"
|
||||
|
||||
#: The synthetic corpus's size — see the module docstring for the
|
||||
#: measured real-world line counts this is modelled on.
|
||||
_CORPUS_COMMITS = 3_000
|
||||
_CORPUS_PATHS_PER_COMMIT = 100
|
||||
|
||||
|
||||
def _corpus_path(commit: int, slot: int) -> str:
|
||||
"""One path line of the synthetic corpus.
|
||||
|
||||
The pool cycles (``(commit * 100 + slot) % 10_000``), so the walk
|
||||
touches 10,000 distinct paths across 300,000 path lines — the same
|
||||
shape as a real repo, where a handful of hot files reappear in
|
||||
commit after commit while history's total path count stays a
|
||||
fraction of the line count.
|
||||
"""
|
||||
return f"path_{(commit * _CORPUS_PATHS_PER_COMMIT + slot) % 10_000:05d}/file_{slot:03d}.md"
|
||||
|
||||
|
||||
def _corpus_stream() -> Iterator[str]:
|
||||
"""The corpus as a LAZY stream: nothing is materialised, each line
|
||||
is built when pulled (the ``stream_git_lines`` shape of task 02)."""
|
||||
for commit in range(_CORPUS_COMMITS):
|
||||
yield _CORPUS_HEADER
|
||||
yield ""
|
||||
for slot in range(_CORPUS_PATHS_PER_COMMIT):
|
||||
yield _corpus_path(commit, slot)
|
||||
|
||||
|
||||
def _corpus_string() -> str:
|
||||
"""The SAME corpus as one big ``str`` (what ``run_git`` returns
|
||||
today) — built OUTSIDE any measured region."""
|
||||
return "\n".join(_corpus_stream()) + "\n"
|
||||
|
||||
|
||||
class _OneShot:
|
||||
"""An iterable that refuses to be iterated twice.
|
||||
|
||||
``__iter__`` raises on the SECOND call, so a parser that
|
||||
materialises its input and rescans it (or loops over ``lines``
|
||||
twice) fails loudly instead of silently passing.
|
||||
"""
|
||||
|
||||
def __init__(self, lines: Iterable[str]) -> None:
|
||||
self._lines = lines
|
||||
self.iter_calls = 0
|
||||
|
||||
def __iter__(self) -> Iterator[str]:
|
||||
self.iter_calls += 1
|
||||
if self.iter_calls > 1:
|
||||
raise AssertionError("re-iterated")
|
||||
return iter(self._lines)
|
||||
|
||||
|
||||
class _CountingStream:
|
||||
"""A stream that records how far the consumer pulled."""
|
||||
|
||||
def __init__(self, lines: list[str]) -> None:
|
||||
self._lines = lines
|
||||
self.pulled = 0
|
||||
|
||||
def __iter__(self) -> Iterator[str]:
|
||||
for pulled, line in enumerate(self._lines, start=1):
|
||||
self.pulled = pulled
|
||||
yield line
|
||||
|
||||
|
||||
def _peak_bytes[T](fn: Callable[[], T]) -> tuple[float, T]:
|
||||
"""Run ``fn`` and return its tracemalloc PEAK in KiB (+ result).
|
||||
|
||||
``reset_peak`` runs after ``start`` so corpus construction (and any
|
||||
earlier test's residue) is excluded — only what ``fn`` itself
|
||||
allocates counts.
|
||||
"""
|
||||
import gc
|
||||
|
||||
gc.collect()
|
||||
tracemalloc.start()
|
||||
try:
|
||||
tracemalloc.reset_peak()
|
||||
result = fn()
|
||||
peak = tracemalloc.get_traced_memory()[1]
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
return peak / 1024, result
|
||||
|
||||
|
||||
def test_parser_accepts_a_line_iterable() -> None:
|
||||
"""Phase 186 D2: the SAME corpus fed as ``iter([...])`` yields the
|
||||
identical dict as the string form — every parsing rule (the
|
||||
``@@%cI`` header, the blank-line skip, the whitespace path split,
|
||||
the ``\\``→``/`` normalisation, the leading-``/`` strip,
|
||||
first-sighting-wins) is untouched."""
|
||||
lines = [
|
||||
"@@2024-06-15T10:00:00+00:00",
|
||||
"",
|
||||
"b.md",
|
||||
"docs\\deep.md /rooted.md",
|
||||
"@@2020-01-02T03:04:06+00:00",
|
||||
"",
|
||||
"a.md",
|
||||
"b.md",
|
||||
]
|
||||
output = "\n".join(lines) + "\n"
|
||||
expected = {
|
||||
"b.md": datetime.fromisoformat("2024-06-15T10:00:00+00:00"),
|
||||
"docs/deep.md": datetime.fromisoformat("2024-06-15T10:00:00+00:00"),
|
||||
"rooted.md": datetime.fromisoformat("2024-06-15T10:00:00+00:00"),
|
||||
"a.md": datetime.fromisoformat("2020-01-02T03:04:06+00:00"),
|
||||
}
|
||||
|
||||
assert _parse_commit_dates(iter(lines)) == expected
|
||||
assert _parse_commit_dates(output) == expected # the string form, unchanged
|
||||
assert _parse_commit_dates(_OneShot(lines)) == expected
|
||||
|
||||
|
||||
def test_parser_rejects_the_malformed_streams_too() -> None:
|
||||
"""Both ``ValueError`` arms fire through the iterable path exactly as
|
||||
through the string path (the caller's fail-soft is unchanged)."""
|
||||
with pytest.raises(ValueError, match="before any commit header"):
|
||||
_parse_commit_dates(iter(["stray.md", "@@2024-06-15T10:00:00+00:00"]))
|
||||
with pytest.raises(ValueError):
|
||||
_parse_commit_dates(iter(["@@yesterday", "b.md"]))
|
||||
|
||||
|
||||
def test_parser_consumes_exactly_once() -> None:
|
||||
"""Phase 186 D4: the parser walks its input in ONE pass.
|
||||
|
||||
The one-shot wrapper raises ``AssertionError("re-iterated")`` on a
|
||||
second ``iter()``; the parse succeeds and the wrapper reports exactly
|
||||
one ``__iter__`` call, so no materialise-then-rescan (and no
|
||||
accidental ``list(lines)`` re-entering the loop) exists. The second
|
||||
half is the stronger laziness pin: a corpus whose offending line
|
||||
sits at index 3 raises there, having pulled EXACTLY 4 lines — a
|
||||
parser that pre-materialised its input would have pulled the whole
|
||||
tail first.
|
||||
"""
|
||||
corpus = [
|
||||
"@@2024-06-15T10:00:00+00:00",
|
||||
"",
|
||||
"b.md",
|
||||
"@@2020-01-02T03:04:06+00:00",
|
||||
"a.md",
|
||||
]
|
||||
one_shot = _OneShot(corpus)
|
||||
assert _parse_commit_dates(one_shot) == {
|
||||
"b.md": datetime.fromisoformat("2024-06-15T10:00:00+00:00"),
|
||||
"a.md": datetime.fromisoformat("2020-01-02T03:04:06+00:00"),
|
||||
}
|
||||
assert one_shot.iter_calls == 1
|
||||
|
||||
counting = _CountingStream(
|
||||
[
|
||||
"@@2024-06-15T10:00:00+00:00",
|
||||
"",
|
||||
"b.md",
|
||||
"@@not-a-date",
|
||||
"b.md",
|
||||
*(["a.md"] * 1_000), # a long tail the parser must never reach
|
||||
]
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
_parse_commit_dates(counting)
|
||||
assert counting.pulled == 4 # stopped AT the bad line, no full read
|
||||
|
||||
|
||||
def test_streamed_parse_alloculates_less_than_buffered() -> None:
|
||||
"""Phase 186 D4: streaming the same corpus allocates materially less
|
||||
than the buffered path, measured comparatively on THIS machine.
|
||||
|
||||
Both arms parse the identical 306,000-line corpus (3,000 commits ×
|
||||
100 paths — see the module docstring for the measured real repos it
|
||||
is modelled on). (a) is the buffered path exactly as ``run_git``
|
||||
serves it today: one big ``str`` that the parser splits
|
||||
(``splitlines()`` materialises a 306,001-entry list of fresh ``str``
|
||||
objects). (b) is a generator building each line lazily. The
|
||||
assertion is ``streamed < buffered * 0.6`` — a wide but not
|
||||
tautological margin (``<=`` would prove nothing); the numbers below
|
||||
are measured on this box (Linux 7.2.9-200.fc44.x86_64, CPython
|
||||
3.13.13, glibc 2.43): buffered **21,504 KiB** peak (the 306,001-entry
|
||||
``splitlines()`` list over a ~7 MB corpus) vs streamed **823 KiB**
|
||||
— a ratio of **0.038**, i.e. the streamed arm allocates ~26× less.
|
||||
The 823 KiB is the retained 10,000-entry result dict, which BOTH
|
||||
arms pay, so the streaming arm is effectively allocation-free.
|
||||
"""
|
||||
corpus = _corpus_string()
|
||||
|
||||
buffered_kb, buffered = _peak_bytes(lambda: _parse_commit_dates(corpus))
|
||||
streamed_kb, streamed = _peak_bytes(lambda: _parse_commit_dates(_corpus_stream()))
|
||||
|
||||
assert buffered_kb > 1_024, f"buffered peak {buffered_kb:,.0f} KiB is not a real corpus"
|
||||
assert streamed_kb < buffered_kb * 0.6, (
|
||||
f"streamed peak {streamed_kb:,.0f} KiB is not materially below the "
|
||||
f"buffered {buffered_kb:,.0f} KiB (ratio "
|
||||
f"{streamed_kb / buffered_kb:.3f}, budget 0.600)"
|
||||
)
|
||||
# memory is not bought by losing data: identical results, ~300k lines
|
||||
assert streamed == buffered
|
||||
assert len(buffered) == 10_000
|
||||
|
||||
|
||||
# --- phase 186: the streaming git site (stream_git_lines) -------------------
|
||||
|
||||
#: The real-git cases below build a throwaway repo (the house ``_git``
|
||||
#: helper) — skipped, not failed, without the git CLI, exactly like
|
||||
#: ``tests/integration/test_git_file_dates.py``.
|
||||
GIT = shutil.which("git") is not None
|
||||
requires_git = pytest.mark.skipif(not GIT, reason="git CLI not available")
|
||||
|
||||
#: Lines spewed to stderr by the deadlock pin — ≈340 KB, i.e. five
|
||||
#: times the ~64 KB pipe buffer, so a non-drained stderr blocks the
|
||||
#: child long before the 10 s wall-clock guard trips.
|
||||
_SPEW_LINES = 60_000
|
||||
|
||||
|
||||
def _git(cwd: Path, *argv: str, when: str | None = None) -> None:
|
||||
"""Run one git command for the test harness (fixture setup); a
|
||||
non-zero exit fails the fixture, not the test under test."""
|
||||
env = os.environ.copy()
|
||||
if when is not None:
|
||||
env["GIT_AUTHOR_DATE"] = when
|
||||
env["GIT_COMMITTER_DATE"] = when
|
||||
env["GIT_AUTHOR_NAME"] = "T"
|
||||
env["GIT_AUTHOR_EMAIL"] = "t@example.com"
|
||||
env["GIT_COMMITTER_NAME"] = "T"
|
||||
env["GIT_COMMITTER_EMAIL"] = "t@example.com"
|
||||
proc = subprocess.run(
|
||||
["git", *argv], cwd=cwd, env=env, capture_output=True, text=True, check=False
|
||||
)
|
||||
assert proc.returncode == 0, f"git {' '.join(argv)} failed: {proc.stderr}"
|
||||
|
||||
|
||||
def _scratch_repo(tmp_path: Path) -> Path:
|
||||
"""A throwaway repo with ONE commit (``a.md``) dated 2020-01-02."""
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
_git(repo, "init", "-q")
|
||||
_git(repo, "config", "user.email", "t@example.com")
|
||||
_git(repo, "config", "user.name", "T")
|
||||
_git(repo, "config", "commit.gpgsign", "false")
|
||||
(repo / "a.md").write_text("# A\n", encoding="utf-8")
|
||||
_git(repo, "add", "-A", when="2020-01-02T03:04:06+00:00")
|
||||
_git(repo, "commit", "-qm", "one", when="2020-01-02T03:04:06+00:00")
|
||||
return repo
|
||||
|
||||
|
||||
class _StreamProc:
|
||||
"""A ``Popen`` stand-in for the abandon-the-stream pin.
|
||||
|
||||
Real pipes would make the case a race (the child has to still be
|
||||
alive at ``close()``); this one holds an unbounded ``StringIO`` so
|
||||
the generator is guaranteed to be parked at its ``yield`` — which is
|
||||
exactly the state whose cleanup the pin checks: terminate, close
|
||||
both pipes, join the stderr thread, reap.
|
||||
"""
|
||||
|
||||
def __init__(self, out: str, err: str) -> None:
|
||||
self.stdout = io.StringIO(out)
|
||||
self.stderr = io.StringIO(err)
|
||||
self.returncode: int | None = None
|
||||
self.terminated = False
|
||||
self.waits = 0
|
||||
|
||||
def terminate(self) -> None:
|
||||
self.terminated = True
|
||||
self.returncode = -15
|
||||
|
||||
def wait(self) -> int:
|
||||
self.waits += 1
|
||||
if self.returncode is None:
|
||||
self.returncode = 0
|
||||
return self.returncode
|
||||
|
||||
|
||||
@requires_git
|
||||
def test_stream_git_lines_yields_lines_and_raises_run_git_shaped_error(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The streaming site's two shapes against REAL git: success yields
|
||||
the ``git log`` lines straight off the pipe (each one still
|
||||
newline-terminated — nothing is buffered or re-split), and a failure
|
||||
raises the ``run_git``-shaped ``GitSyncError`` (``git <subcommand>
|
||||
failed (exit N): <stderr>``), so every caller's fail-soft text is
|
||||
unchanged."""
|
||||
repo = _scratch_repo(tmp_path)
|
||||
argv = ["git", "log", "--name-only", "--format=@@%cI"]
|
||||
|
||||
lines = list(stream_git_lines(argv, cwd=repo))
|
||||
|
||||
stripped = [line.rstrip("\n") for line in lines]
|
||||
assert stripped[1:] == ["", "a.md"]
|
||||
assert stripped[0].startswith("@@") # the %cI commit header, unsplit
|
||||
# (%cI prints UTC as "Z"; fromisoformat reads both forms)
|
||||
assert datetime.fromisoformat(stripped[0][2:]) == datetime(2020, 1, 2, 3, 4, 6, tzinfo=UTC)
|
||||
assert all(line.endswith("\n") for line in lines)
|
||||
|
||||
plain = tmp_path / "not-a-repo"
|
||||
plain.mkdir()
|
||||
with pytest.raises(
|
||||
GitSyncError, match=r"^git log --name-only --format=@@%cI failed \(exit \d+\): "
|
||||
) as excinfo:
|
||||
list(stream_git_lines(argv, cwd=plain))
|
||||
assert "not a git repository" in str(excinfo.value)
|
||||
|
||||
|
||||
@requires_git
|
||||
def test_stream_git_lines_reports_stderr_without_deadlocking(tmp_path: Path) -> None:
|
||||
"""A2: a child spilling ≈340 KB on stderr while stdout is being
|
||||
consumed neither deadlocks nor truncates.
|
||||
|
||||
The command is a git alias running a shell that writes
|
||||
``seq 1 60_000`` (≈340 KB — five pipe buffers) to stderr and exits
|
||||
3, so git's own stderr pipe is what must be drained. Without the
|
||||
drain thread the child blocks on a full stderr pipe while the parent
|
||||
blocks on stdout and neither ever finishes (the classic two-pipe
|
||||
``Popen`` trap); here the call returns in well under the 10 s guard
|
||||
and the ``GitSyncError`` carries git's FULL stderr, which is why
|
||||
``stderr=DEVNULL`` is not the answer (and ``stderr=STDOUT`` would
|
||||
corrupt the date parse).
|
||||
"""
|
||||
repo = _scratch_repo(tmp_path)
|
||||
argv = ["git", "-c", f'alias.spew=!sh -c "seq 1 {_SPEW_LINES} >&2; exit 3"', "spew"]
|
||||
|
||||
started = time.monotonic()
|
||||
with pytest.raises(GitSyncError) as excinfo:
|
||||
list(stream_git_lines(argv, cwd=repo))
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
msg = str(excinfo.value)
|
||||
assert elapsed < 10, f"streaming a chatty stderr took {elapsed:.1f}s — deadlock?"
|
||||
assert "(exit 3)" in msg
|
||||
assert len(msg.encode()) > 100_000, "git's stderr was truncated, not drained"
|
||||
assert msg.rstrip().endswith(str(_SPEW_LINES)) # the LAST line is there
|
||||
|
||||
|
||||
def test_stream_git_lines_terminates_an_abandoned_stream(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Closing the generator early (``GeneratorExit``) — and an exception
|
||||
thrown into it — never leaks a ``git`` child: the process is
|
||||
terminated, both pipes closed, the drain thread joined, the child
|
||||
reaped."""
|
||||
proc = _StreamProc("".join(f"line {i}\n" for i in range(5_000)), "warning: noisy stderr\n")
|
||||
monkeypatch.setattr(subprocess, "Popen", lambda *a, **k: proc)
|
||||
|
||||
# The public shape is ``Iterator[str]`` (the parser's contract); this
|
||||
# pin exercises the generator protocol behind it — close()/throw().
|
||||
gen = cast(
|
||||
"Generator[str, None, None]",
|
||||
stream_git_lines(["git", "log", "--name-only", "--format=@@%cI"], cwd=tmp_path),
|
||||
)
|
||||
assert next(gen) == "line 0\n" # parked at the yield, 4,999 lines unread
|
||||
gen.close()
|
||||
|
||||
assert proc.terminated is True
|
||||
assert proc.returncode == -15 # wait() reaped it — no zombie, no orphan
|
||||
assert proc.waits >= 1
|
||||
assert proc.stdout.closed and proc.stderr.closed
|
||||
|
||||
# the same arm fires for a consumer that dies on its own exception
|
||||
proc2 = _StreamProc("line 0\nline 1\n", "")
|
||||
monkeypatch.setattr(subprocess, "Popen", lambda *a, **k: proc2)
|
||||
gen2 = cast("Generator[str, None, None]", stream_git_lines(["git", "log"], cwd=tmp_path))
|
||||
assert next(gen2) == "line 0\n"
|
||||
with pytest.raises(RuntimeError, match="the parser blew up"):
|
||||
gen2.throw(RuntimeError("the parser blew up"))
|
||||
assert proc2.terminated is True
|
||||
|
||||
|
||||
def test_stream_git_lines_missing_git_raises_named_error(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A missing git binary is the same ``GitSyncError`` as through
|
||||
:func:`run_git` — the walk's fail-soft never sees a bare
|
||||
``FileNotFoundError``."""
|
||||
|
||||
def boom(*args: object, **kwargs: object) -> None:
|
||||
raise FileNotFoundError("git")
|
||||
|
||||
monkeypatch.setattr(subprocess, "Popen", boom)
|
||||
|
||||
with pytest.raises(GitSyncError, match="git was not found on PATH"):
|
||||
list(stream_git_lines(["git", "log"], cwd=tmp_path))
|
||||
# …and the walk's fail-soft keeps working on it: {} + the mtime
|
||||
# fallback, never a bare FileNotFoundError out of the sync.
|
||||
assert git_sync.file_commit_dates(tmp_path) == {}
|
||||
|
||||
|
||||
def test_file_commit_dates_fails_soft_on_a_mid_stream_git_failure(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Phase 186 D3: a failure AFTER some lines were parsed still yields
|
||||
``{}`` + the same warning (the D2 contract is about the RESULT).
|
||||
|
||||
Pre-streaming, ``run_git`` could only fail before a single line was
|
||||
seen; the stream can fail half-way through a huge history. The
|
||||
importer must still fall back to mtimes — a date walk must never
|
||||
break a sync — and the partial map must be DROPPED, not returned.
|
||||
"""
|
||||
def half_then_boom(argv: list[str], cwd: Path) -> Iterator[str]:
|
||||
yield "@@2024-06-15T10:00:00+00:00"
|
||||
yield "a.md"
|
||||
raise GitSyncError("git log --name-only --format=@@%cI failed (exit 128): fatal: boom")
|
||||
|
||||
monkeypatch.setattr(git_sync, "stream_git_lines", half_then_boom)
|
||||
with caplog.at_level(logging.WARNING, logger="scripts.git_sync"):
|
||||
assert git_sync.file_commit_dates(tmp_path) == {}
|
||||
warnings = [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert warnings[0].startswith(f"file_commit_dates: git log failed for {tmp_path}: ")
|
||||
assert "fall back to file mtimes" in warnings[0]
|
||||
|
||||
|
||||
#: Child-process spawns by name (``subprocess.run``/``Popen``/… and the
|
||||
#: ``os`` equivalents) — what an A11 audit has to catch, not just the
|
||||
#: literal string "subprocess".
|
||||
_SUBPROCESS_SPAWNS = frozenset(
|
||||
{"Popen", "run", "call", "check_call", "check_output", "getoutput", "getstatusoutput"}
|
||||
)
|
||||
_OS_SPAWNS = frozenset(
|
||||
{"system", "popen", "execl", "execle", "execlp", "execv", "execve", "execvp", "posix_spawn"}
|
||||
)
|
||||
|
||||
|
||||
class _SpawnVisitor(ast.NodeVisitor):
|
||||
"""Collects ``(enclosing function, spawn call)`` for every child process."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.stack: list[str] = []
|
||||
self.sites: list[tuple[str, str]] = []
|
||||
|
||||
@property
|
||||
def _scope(self) -> str:
|
||||
return ".".join(self.stack) if self.stack else "<module>"
|
||||
|
||||
def _visit_func(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> None:
|
||||
self.stack.append(node.name)
|
||||
self.generic_visit(node)
|
||||
self.stack.pop()
|
||||
|
||||
visit_FunctionDef = _visit_func
|
||||
visit_AsyncFunctionDef = _visit_func
|
||||
|
||||
def visit_Call(self, node: ast.Call) -> None:
|
||||
fn = node.func
|
||||
if isinstance(fn, ast.Attribute) and isinstance(fn.value, ast.Name):
|
||||
if fn.value.id == "subprocess" and fn.attr in _SUBPROCESS_SPAWNS:
|
||||
self.sites.append((self._scope, f"subprocess.{fn.attr}"))
|
||||
elif fn.value.id == "os" and fn.attr in _OS_SPAWNS:
|
||||
self.sites.append((self._scope, f"os.{fn.attr}"))
|
||||
self.generic_visit(node)
|
||||
|
||||
|
||||
def _spawn_sites(path: Path) -> list[tuple[str, str]]:
|
||||
"""Every real spawn in one module, tagged with its function.
|
||||
|
||||
Parsing the AST (not grepping text) keeps docstrings that MENTION
|
||||
``subprocess.run`` out of the result — this module has several.
|
||||
"""
|
||||
visitor = _SpawnVisitor()
|
||||
visitor.visit(ast.parse(path.read_text(encoding="utf-8")))
|
||||
return visitor.sites
|
||||
|
||||
|
||||
def test_only_two_git_spawn_sites() -> None:
|
||||
"""A11, phase-186 shape: exactly TWO spawn sites, one per shape,
|
||||
both inside ``scripts/git_sync.py`` — the buffered
|
||||
``run_git`` (``subprocess.run``) and the streaming
|
||||
``stream_git_lines`` (``subprocess.Popen``).
|
||||
|
||||
The repo-wide half asserts no NEW spawn appears in ``app/`` or
|
||||
``scripts/``. It enumerates the ONE pre-existing spawn outside the
|
||||
module: ``app/core/caching.py``'s boot-time asset-version probe (a
|
||||
5-second-timeouted ``git rev-parse --short HEAD`` that never raises
|
||||
and is not part of any sync path) — the honest state of A11, which
|
||||
the module docstring now states too, instead of a pin that quietly
|
||||
ignores it. ``app/core/docs_push.py`` passes by construction: it
|
||||
calls ``run_git``, so it never appears here.
|
||||
"""
|
||||
module = Path(git_sync.__file__)
|
||||
|
||||
assert _spawn_sites(module) == [
|
||||
("run_git", "subprocess.run"),
|
||||
("stream_git_lines", "subprocess.Popen"),
|
||||
]
|
||||
|
||||
repo_root = module.parents[1]
|
||||
offenders: dict[str, list[tuple[str, str]]] = {}
|
||||
for package in ("app", "scripts"):
|
||||
for py in sorted((repo_root / package).rglob("*.py")):
|
||||
if py == module:
|
||||
continue
|
||||
found = _spawn_sites(py)
|
||||
if found:
|
||||
offenders[str(py.relative_to(repo_root))] = found
|
||||
assert offenders == {"app/core/caching.py": [("_git_short_sha", "subprocess.run")]}
|
||||
|
||||
# docs-push rides the buffered site rather than spawning its own —
|
||||
# the reason it is absent from `offenders`.
|
||||
assert docs_push.run_git is git_sync.run_git
|
||||
|
||||
Reference in New Issue
Block a user