phase: 186_git_log_streaming_dates
## Final report — phase 186 `git_log_streaming_dates`
**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.
### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |
Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.
### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.
Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.
⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).
Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
This commit is contained in:
@@ -26,6 +26,17 @@ the VERIFIED checkout behavior (phase 107):
|
||||
``{}`` — a date walk must never break a sync (the importer, task
|
||||
04, falls back to file mtimes).
|
||||
|
||||
Phase 186 adds the two pins for the walk moving off ``run_git`` onto
|
||||
the streaming site :func:`scripts.git_sync.stream_git_lines` (the
|
||||
incident of 2026-10-15: a full-history ``git log`` held as ONE string
|
||||
plus the parser's ``splitlines()`` copy, tens of MB of transient peak
|
||||
per source): ``test_streamed_walk_returns_the_buffered_dates`` proves
|
||||
the streamed dict EQUALS the buffered one on the four cases that
|
||||
matter (commit-1-only file, commit-2-modified file, subdirectory file,
|
||||
never-committed file → the mtime fallback), and
|
||||
``test_file_commit_dates_fail_soft_unchanged`` pins that both fail-soft
|
||||
arms and their exact warning lines survived the move.
|
||||
|
||||
DB-free by design: ``file_commit_dates`` takes a path, no session.
|
||||
Skipped (not failed) on a machine without the git CLI (the
|
||||
``test_doc_drafts_api.py`` guard).
|
||||
@@ -212,12 +223,13 @@ def test_git_error_fails_soft(
|
||||
) -> None:
|
||||
"""(c) a ``GitSyncError`` from the walk (git missing/failed) →
|
||||
``{}`` + a logged warning naming the fallback — the fail-soft
|
||||
contract (pinned)."""
|
||||
contract (pinned). Phase 186 moved the walk off ``run_git`` onto
|
||||
:func:`stream_git_lines`, so that is the site the stub replaces."""
|
||||
|
||||
def boom(argv: list[str], cwd: Path) -> str:
|
||||
raise GitSyncError("git log failed (exit 128): fatal: bad object")
|
||||
|
||||
monkeypatch.setattr("scripts.git_sync.run_git", boom)
|
||||
monkeypatch.setattr("scripts.git_sync.stream_git_lines", boom)
|
||||
with caplog.at_level("WARNING"):
|
||||
assert file_commit_dates(scratch_repo) == {}
|
||||
assert any("file_commit_dates" in record.message for record in caplog.records)
|
||||
@@ -230,12 +242,88 @@ def test_malformed_log_output_fails_soft(
|
||||
read) → ``{}`` (fail-soft) — the walk is all-or-nothing: a
|
||||
partially parsed date map would be worse than none."""
|
||||
monkeypatch.setattr(
|
||||
"scripts.git_sync.run_git",
|
||||
lambda argv, cwd: "@@not-a-date\nb.md\n",
|
||||
"scripts.git_sync.stream_git_lines",
|
||||
lambda argv, cwd: iter(["@@not-a-date", "b.md"]),
|
||||
)
|
||||
assert file_commit_dates(scratch_repo) == {}
|
||||
|
||||
|
||||
# --- phase 186: the streamed walk returns what the buffered one did ---------
|
||||
|
||||
|
||||
def test_streamed_walk_returns_the_buffered_dates(
|
||||
scratch_repo: Path, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 186 D4 — DATE PARITY. The same checkout is walked twice:
|
||||
through the shipped streaming path (``file_commit_dates``) and
|
||||
through the pre-phase buffered path (``run_git`` → one ``str`` →
|
||||
``_parse_commit_dates``). The two dicts are EQUAL, and they carry
|
||||
the four cases the phase names:
|
||||
|
||||
(i) a file only in commit 1 (``a.md``) → commit 1's date;
|
||||
(ii) a file modified in commit 2 (``b.md``) → commit 2's date, NOT
|
||||
commit 1's — first-sighting-wins over a newest-first walk;
|
||||
(iii) a file in a subdirectory (``docs/deep.md``) → its own
|
||||
last-commit date, keyed repo-relative with ``/``;
|
||||
(iv) a file in the working tree that git never committed → no entry
|
||||
at all, so the importer's mtime fallback applies (asserted on
|
||||
the LOOKUP here — the resulting date is the importer's
|
||||
contract, tested in ``test_importer_dates.py``)."""
|
||||
dest = tmp_path / "clone"
|
||||
clone_or_pull(str(scratch_repo), dest)
|
||||
(dest / "never-committed.md").write_text("# untracked\n", encoding="utf-8")
|
||||
|
||||
streamed = file_commit_dates(dest)
|
||||
buffered = _parse_commit_dates(
|
||||
run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)
|
||||
)
|
||||
|
||||
assert streamed == buffered
|
||||
assert streamed["a.md"] == DATE_A # (i) only in commit 1
|
||||
assert streamed["b.md"] == DATE_B # (ii) modified in commit 2
|
||||
assert streamed["docs/deep.md"] == DATE_A # (iii) subdirectory
|
||||
assert "never-committed.md" not in streamed # (iv) mtime fallback
|
||||
assert streamed.get("never-committed.md") is None
|
||||
|
||||
|
||||
def test_file_commit_dates_fail_soft_unchanged(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Phase 186 D3: both fail-soft arms and their EXACT warning lines
|
||||
survive the switch to streaming — (a) a checkout that is not a
|
||||
directory (git is never invoked), (b) a directory that is not a
|
||||
repository (the stream raises the ``run_git``-shaped
|
||||
``GitSyncError`` mid-call). Both return ``{}``; neither raises.
|
||||
The wording is byte-pinned because the ops dashboards grep it."""
|
||||
missing = tmp_path / "gone"
|
||||
not_a_dir = tmp_path / "a-file"
|
||||
not_a_dir.write_text("# not a directory\n", encoding="utf-8")
|
||||
plain = tmp_path / "notarepo"
|
||||
plain.mkdir()
|
||||
(plain / "a.md").write_text("# A\n", encoding="utf-8")
|
||||
|
||||
with caplog.at_level("WARNING", logger="scripts.git_sync"):
|
||||
assert file_commit_dates(missing) == {}
|
||||
assert file_commit_dates(not_a_dir) == {}
|
||||
assert file_commit_dates(plain) == {}
|
||||
|
||||
messages = [r.getMessage() for r in caplog.records if r.name == "scripts.git_sync"]
|
||||
assert len(messages) == 3
|
||||
assert messages[0] == (
|
||||
f"file_commit_dates: {missing} is not a directory — no git dates "
|
||||
"(the importer will fall back to file mtimes)"
|
||||
)
|
||||
assert messages[1] == (
|
||||
f"file_commit_dates: {not_a_dir} is not a directory — no git dates "
|
||||
"(the importer will fall back to file mtimes)"
|
||||
)
|
||||
assert messages[2].startswith(f"file_commit_dates: git log failed for {plain}: git log ")
|
||||
assert "failed (exit " in messages[2]
|
||||
assert messages[2].endswith(
|
||||
"— the importer will fall back to file mtimes"
|
||||
)
|
||||
|
||||
|
||||
# --- the pure parser (canned git output — no git, no DB) -------------------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user