phase: 183_sync_passive_delete_cascade
All gates green. Final state verified.
**Implemented / verified**
- Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after.
- Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings.
- **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`).
**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%)
- `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
**Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands).
**Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit).
**Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify.
**Next pending phase**: `184_sync_bounded_session`.
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# Phase 186 — The commit-date walk streams instead of buffering the whole `git log`
|
||||
|
||||
**Source:** owner incident 2026-10-15 (as phases 183–185) — `_run_sync` builds `doc_dates_by_root` from a full-history `git log --name-only` captured ENTIRELY in memory, per source, and keeps the result for the whole run.
|
||||
**Story:** n/a (incident-derived, owner directive 2026-10-15)
|
||||
**Depends on:** none functionally; sequenced after 183–185 so the sync's memory work lands together.
|
||||
|
||||
**Context:** `scripts/git_sync.py` — the module docstring's **A11 "every git command goes through `run_git`" claim at L31**, `clone_or_pull` (L96), `_unshallow_if_shallow` (L221), **`_parse_commit_dates(output: str)` (L240)** — `for line in output.splitlines()` at L245-ish, the first-sighting-wins `dates.setdefault(path, commit)`, the `ValueError` paths; **`file_commit_dates` (L270)** — `output = run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)` (L298), the two fail-soft `except` arms returning `{}` with the `file_commit_dates: …` warnings, `run_git` (**L320**, `subprocess.run(argv, cwd=cwd, capture_output=True, text=True, env=env)` at **L342**, `GitSyncError` with `proc.stderr` on non-zero); `app/api/sync.py` — `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)` (**L646**, phase 179), the dict's consumer `dates_map = (doc_dates_by_root or {}).get(str(root), {})` in `app/rag/importer.py` (**L913**), `raw_date=dates_map.get(rel)` passed to `_index_file` (**L945–948**), the mtime fallback `file_mtime_datetime` (`app/rag/doc_dates.py` L57) and `normalize_doc_date`; `app/rag/doc_dates.py` (pure, unchanged); `tests/unit/test_git_sync.py` (the `_parse_commit_dates` + argv pins, which call it with **strings**), `tests/integration/test_git_file_dates.py`, `tests/integration/test_importer_dates.py`, `tests/e2e/test_git_source_dates.py`, `tests/e2e/test_document_dates.py`.
|
||||
|
||||
## Measured baseline (2026-10-15, the incident box's own checkouts, read-only `git` inside the app container)
|
||||
| checkout | commits | `git log --name-only --format=@@%cI` lines | bytes |
|
||||
|---|---|---|---|
|
||||
| litellm | 54,502 | 346,829 | 15,077,351 |
|
||||
| llama.cpp | 11,659 | 69,922 | 1,867,046 |
|
||||
| vllm-radiance | 192 | 4,855 | 387,672 |
|
||||
| gufo | 589 | 9,155 | 327,032 |
|
||||
|
||||
`run_git` returns that as ONE `str`; `_parse_commit_dates` then calls `.splitlines()`, materialising a second copy as a list of 346,829 `str` objects — tens of MB of transient peak per source on a 3,904 MB box, before the retained dict. The source removed that morning (`server`, nextcloud) is the case that shows how unbounded this is: the walk scales with the repository's ENTIRE history, not with its working tree.
|
||||
|
||||
## Objective
|
||||
Stream the walk: parse the `git log` output line by line from the subprocess pipe so the run never holds the whole output, keep the phase-106 D2 contract byte-for-byte (first-sighting-wins last-commit dates, the same fail-soft arms, the same warning lines, mtime fallback), and amend the A11 single-site claim so the streaming variant is a documented, tested sibling rather than an exception hiding in the module.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **D1 — `stream_git_lines(argv, cwd)` is the second git site, and A11 says so (task 02).** A generator in `scripts/git_sync.py`:
|
||||
```python
|
||||
def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
|
||||
"""Stream one git command's stdout, line by line (phase 186, A11 sibling)."""
|
||||
```
|
||||
`subprocess.Popen(argv, cwd=cwd, stdout=PIPE, stderr=PIPE, text=True)`; stdout is iterated directly (`for line in proc.stdout`); **stderr is drained by a daemon thread into a list** so a chatty stderr can never deadlock the pipe while stdout is being consumed (the classic `Popen` + two-pipes trap — this is the reason not to `communicate()` at the end and not to `stderr=STDOUT`, which would pollute the parse). On `proc.wait() != 0` it raises `GitSyncError(f"git {' '.join(argv[1:])} failed (exit …): {stderr}")` — the SAME message shape as `run_git` (L344–347), so every caller's fail-soft handling is unchanged. The module docstring's L31 claim becomes: `run_git` is the buffered site, `stream_git_lines` the streaming one, and **both** are the only places a `subprocess` is spawned (a unit pin greps `subprocess.` occurrences in the file — the phase-179 AGENTS rule 11 "one git-invocation site" intent survives as "one for each shape, both owned here").
|
||||
- **D2 — the parser takes any iterable; strings keep working (task 01 first).** `_parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]`: `if isinstance(lines, str): lines = lines.splitlines()` and everything else — the `@@%cI` commit-header rule, the whitespace path split, the `\`→`/` normalisation, the leading-`/` strip, `setdefault` first-sighting-wins, both `ValueError` arms — stays byte-identical, so `tests/unit/test_git_sync.py`'s existing string pins need no edits.
|
||||
- **D3 — `file_commit_dates` consumes the stream, fail-soft unchanged (task 02).** `return _parse_commit_dates(stream_git_lines([...], cwd=dest))` inside the existing `try`, the existing `except GitSyncError` → the same warning + `{}`, the existing `except ValueError` → the same warning + `{}`. Two consequences documented in the docstring, not hidden: a mid-stream failure now surfaces AFTER some lines were parsed, and the method still returns `{}` because the caller falls back to mtimes (the phase-106 D2 fail-soft contract is about the RESULT, and the result is unchanged); and the returned dict — one entry per path in the repository's history — remains retained by `_run_sync` in `doc_dates_by_root` for the run (A3: out of scope; the dict is ~10% of the buffered cost this phase removes, and bounding it changes dates, not just memory).
|
||||
- **D4 — the pins are comparative and deterministic (tasks 01 + 02).** Unit: a one-shot generator input proves single-pass consumption (a generator that raises on a second `iter()`), and the lazy-vs-buffered assertion measures `tracemalloc` peak for the SAME synthetic corpus (≈300k lines) through (a) one big `str` + `splitlines()` and (b) a generator, asserting `streamed < buffered * 0.6` with both numbers printed in the failure message. Integration: `tests/integration/test_git_file_dates.py` extended with a generated two-commit repo proving the post-fix dates equal the pre-fix dates for (i) a file only in commit 1, (ii) a file modified in commit 2 (its date is commit 2's), (iii) a path in a subdirectory, (iv) a file absent from history entirely (mtime fallback).
|
||||
- **D5 — the E2E pins the owner-visible date (task 03).** `tests/e2e/test_git_log_stream_dates.py`: a local `file://` fixture repo with two dated commits (the `test_git_source_dates.py` `_git` helper shape, `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` pinned so the dates are exact), synced from the Sources page; the document date shown for the modified file equals commit 2's date and the untouched file shows commit 1's date (the phase-106 contract, now through the stream), a file git never committed falls back to its mtime, and the D1 manual-date lock still survives a re-sync. Whatever the walk currently does with a quoted non-ASCII path is recorded in the module docstring as-is and pinned as UNCHANGED behaviour (this phase does not change the parser's quoting semantics — it only changes how the bytes arrive).
|
||||
- **NOT touched:** the `git log` ARGV (`--name-only --format=@@%cI` stays, pinned by the existing argv unit tests), `clone_or_pull` / `_unshallow_if_shallow` (they stay on `run_git`), `app/rag/doc_dates.py` (pure), the importer's `dates_map.get(rel)` lookup, the mtime fallback, the sync's cancel semantics (phase 134/179), the frontend.
|
||||
|
||||
## Dependencies
|
||||
— (none functionally; run after `183`/`184`/`185`)
|
||||
|
||||
## Tasks
|
||||
1. `01_parser_accepts_lines.md` — `_parse_commit_dates` takes `str | Iterable[str]` with byte-identical semantics, the one-shot-generator + comparative-memory unit pins, the existing string pins untouched.
|
||||
2. `02_stream_git_log.md` — `stream_git_lines` (Popen + stderr thread + the `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream, the A11 docstring amendment, the grep pin, the integration date-parity tests.
|
||||
3. `03_e2e_streamed_dates.md` — `tests/e2e/test_git_log_stream_dates.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
- **Unit:** `tests/unit/test_git_sync.py` — existing string pins green unedited; the new iterable/one-shot/lazy-memory pins; the `subprocess.` occurrence grep pin.
|
||||
- **Integration:** `tests/integration/test_git_file_dates.py` (date parity on a generated repo, the fail-soft arms still returning `{}` with the same warnings), `tests/integration/test_importer_dates.py`, `test_import_docs_git.py`, `test_sync_api.py` (the `file_commit_dates` monkeypatch points keep working — `app.api.sync` imports the name, phase 179's probe still patches it).
|
||||
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_git_log_stream_dates.py`, run **in isolation**.
|
||||
- **Coverage:** **>90%** on `app/` plus full coverage of `scripts/git_sync.py`'s new function (the existing `tests/unit/test_git_sync.py` family covers this module today — keep it covered, not just `app/`).
|
||||
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Documentation & Consistency
|
||||
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
|
||||
- `AGENTS.md` rule 11 (the "never block the event loop" rule already cites `run_git`): one clause noting the streaming sibling exists and that BOTH are off-loop via `asyncio.to_thread` (phase 179) — do not rewrite the rule, extend the reference. `README.md`: nothing user-facing changes; add the ops note that the commit-date walk streams (so a huge repo no longer needs the whole `git log` in RAM) and that the per-source date dict is still held for the run (A3).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `_parse_commit_dates` accepts `str | Iterable[str]` with byte-identical parse semantics; the existing string-based unit pins are green with no edits.
|
||||
- [ ] `stream_git_lines` exists, drains stderr on a thread, raises the `run_git`-shaped `GitSyncError`; `file_commit_dates` streams; the module docstring's A11 claim names both sites and the grep pin enforces it.
|
||||
- [ ] The lazy-vs-buffered memory comparison and the one-shot generator pins are green with their measured numbers in the test docstrings.
|
||||
- [ ] `uv run pytest tests/integration/test_git_file_dates.py tests/unit/test_git_sync.py -v` green (date parity across all four cases, both fail-soft arms unchanged).
|
||||
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` green **in isolation**.
|
||||
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
|
||||
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(sync): stream the git commit-date walk instead of buffering git log in RAM`); the phase dir moves to `.agents/phases/complete/`.
|
||||
|
||||
## Locked decisions (owner-confirmed 2026-10-15)
|
||||
- **A1 — stream, do not shrink the argv.** No `--since`, no `--max-count`, no per-file `git log -1`: every one of those changes WHICH dates the KB stores, and the phase-106 D2 last-commit-date contract is a data contract, not a performance knob.
|
||||
- **A2 — stderr on a worker thread.** The deadlock-free way to stream stdout while still reporting git's stderr; `stderr=STDOUT` would corrupt the parse, `stderr=DEVNULL` would lose the error text `GitSyncError` carries today.
|
||||
- **A3 — the retained `doc_dates_by_root` dict is out of scope.** Bounding it (by age, by working-tree membership, or by streaming it to a temp file) changes document dates or adds a new subsystem; this phase removes the transient whole-`git log` copy, which is the unbounded piece.
|
||||
- **A4 — `run_git` keeps its shape.** Every other caller (`clone_or_pull`, `_unshallow_if_shallow`, `app/core/docs_push.py`) keeps the buffered site; migrating them is churn without a measured cost.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 01 — The parser takes a stream (semantics untouched)
|
||||
|
||||
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Widen `_parse_commit_dates` to accept any iterable of lines (keeping `str` working so every existing pin is untouched), and pin in tests that it consumes its input exactly once and that streaming allocates materially less than the buffered path for the same corpus.
|
||||
|
||||
## Work
|
||||
1. `scripts/git_sync.py` — `_parse_commit_dates` (**L240**): signature `def _parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]:` with the single line `if isinstance(lines, str): lines = lines.splitlines()` at the top; rename the loop variable to match; `from collections.abc import Iterable` added. Every parsing rule stays byte-identical: the `@@`-prefixed `%cI` header (`datetime.fromisoformat(line[2:])`), the blank-line skip, `line.split()` for multiple paths per line, `raw_path.replace("\\", "/").lstrip("/")`, `dates.setdefault(path, commit)` (newest-first, first sighting wins), and BOTH `ValueError` arms (a malformed date, a path line before any commit header). The docstring gains: "*lines* may be a string or any iterable of lines (phase 186 — `stream_git_lines` feeds it a live pipe); a string input behaves exactly as before."
|
||||
2. `tests/unit/test_git_sync.py` — extend (existing string-based tests untouched):
|
||||
- `test_parser_accepts_a_line_iterable`: the same corpus the existing tests use, fed as `iter([...])`, produces the identical dict.
|
||||
- `test_parser_consumes_exactly_once`: a one-shot generator wrapper that raises `AssertionError("re-iterated")` if `iter()` is called twice on it; parse succeeds → proves the parser does not materialise-then-rescan (and rules out an accidental `list(lines)` inside the loop).
|
||||
- `test_streamed_parse_alloculates_less_than_buffered`: build ONE synthetic corpus of ~300,000 lines in the exact real shape (3,000 commits × 100 paths, `@@2026-01-01T00:00:00+00:00` headers), then measure `tracemalloc` peak over (a) `_parse_commit_dates(one_big_str)` (the buffered path's own `splitlines()`) and (b) `_parse_commit_dates(iter_of_lines_from_a_generator)` (each line built lazily from a fixed seed), asserting `streamed_peak < buffered_peak * 0.6` with both numbers in the assertion message. Record both numbers + the machine in the docstring; if the real ratio on this box differs, keep the 0.6 factor only if the pre/post split genuinely clears it and document the measured ratio (never weaken to `<= buffered_peak`, which is a tautology).
|
||||
- Corpus justification in the module docstring (not a test): the incident box's measured sizes — litellm 54,502 commits / 346,829 lines / 15 MB; llama.cpp 69,922 lines; vllm-radiance 4,855; gufo 9,155 — which is why the synthetic corpus is ~300k lines.
|
||||
3. Verify: `uv run pytest tests/unit/test_git_sync.py -v`; `uv run pytest` green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the three new pins + every existing pin green unedited.
|
||||
- Coverage: **>90%** on `app/` and `scripts/git_sync.py`'s parser stays fully covered.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `_parse_commit_dates` accepts `str | Iterable[str]`; no parsing rule changed; the docstring says so.
|
||||
- [ ] The one-shot-generator pin is green (no re-iteration, no hidden `list()`).
|
||||
- [ ] The streamed-vs-buffered memory comparison is green with both measured peaks recorded in the docstring.
|
||||
- [ ] `uv run pytest tests/unit/test_git_sync.py -v` and `uv run pytest` green.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Task 02 — The fix: `stream_git_lines` + `file_commit_dates` off the buffer
|
||||
|
||||
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Add the streaming git site and put the commit-date walk on it, so no `git log` output is ever held as one string — with `run_git`'s error contract, the phase-106 D2 fail-soft result, and the A11 single-site rule kept intact (as two documented siblings).
|
||||
|
||||
## Work
|
||||
1. `scripts/git_sync.py` — new function next to `run_git` (L320):
|
||||
```python
|
||||
def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
|
||||
```
|
||||
- `proc = subprocess.Popen(argv, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=None)` (the same env semantics as `run_git`: `None` inherits — do NOT introduce the env-replacement trap the module docstring warns about).
|
||||
- stderr drained by a daemon thread appending to a list, started before stdout is consumed (A2: the deadlock-free shape — never `communicate()` after iterating stdout, never `stderr=STDOUT`, never `stderr=DEVNULL`).
|
||||
- `for line in proc.stdout or (): yield line` (guard the `text=True` optional type for pyright), then `proc.stdout.close()`, `err_thread.join()`, `rc = proc.wait()`; `if rc != 0: raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {rc}): " + "".join(stderr_lines).strip())` — the message shape copied from `run_git` L344–347 so every caller's fail-soft text is unchanged.
|
||||
- `finally`-safe: if the generator is closed early (`GeneratorExit`), terminate the process (`proc.terminate()` guarded by try/except) so a partial consume never leaks a `git` child.
|
||||
- Module docstring: amend the **L31** claim → `run_git` (buffered) and `stream_git_lines` (streaming, phase 186) are the ONLY two places a git subprocess is spawned; both own the argv→`GitSyncError` contract; both are reached from async code only through `asyncio.to_thread` (phase 179 / AGENTS rule 11).
|
||||
2. `file_commit_dates` (**L270**): `output = run_git([...])` (L298) → `return _parse_commit_dates(stream_git_lines(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest))` inside the existing `try`, keeping BOTH `except` arms and their exact warning strings. Add the docstring notes: the walk streams (no whole-`git log` in RAM — the incident's transient peak); a mid-stream failure still yields `{}` + the mtime fallback (the D2 contract is about the RESULT); and the returned per-source dict stays retained by `_run_sync` for the whole run — **A3, out of scope, named**.
|
||||
3. `tests/unit/test_git_sync.py` — add:
|
||||
- `test_stream_git_lines_yields_lines_and_raises_run_git_shaped_error` — run it against a real throwaway git repo (the house `_git` helper): success path yields the expected lines; `git log` in a non-repo dir raises `GitSyncError` whose message matches `^git log failed \(exit \d+\): ` (the `run_git` shape).
|
||||
- `test_stream_git_lines_reports_stderr_without_deadlocking` — a command that writes >100 KB to stderr (e.g. `git -c core.quotepath=bogus …` is contrived; use `git log` in a repo with a corrupt arg that spews usage, or `git` with `--help` on stderr via a bad flag) and assert the call raises with a non-empty stderr excerpt and completes in a bounded time (a wall-clock guard, e.g. `< 10 s`).
|
||||
- `test_only_two_git_spawn_sites` — read `scripts/git_sync.py` and assert `subprocess.` occurrences are exactly inside `run_git` and `stream_git_lines` (the A11 grep pin), and that no other module in `app/` or `scripts/` spawns `subprocess` for git (`grep`-style assertion over the repo; if the docs-push path legitimately uses `run_git`, it passes by construction).
|
||||
- Keep every existing `run_git`/argv pin green (the `git log` argv is unchanged — A1).
|
||||
4. `tests/integration/test_git_file_dates.py` — extend with the date-parity fixture (a generated repo, `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` pinned per commit): file only in commit 1 → commit 1's date; file modified in commit 2 → commit 2's date; a file in a subdirectory → its own last-commit date; a file that exists but was never committed → `{}` entry, so the importer's mtime fallback applies (assert the empty lookup, not the resulting date, to keep this test about `file_commit_dates`). Also `test_file_commit_dates_fail_soft_unchanged`: a non-directory and a non-repo directory both return `{}` with the existing warning lines (wording pinned).
|
||||
5. Verify: `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v`; then `uv run pytest`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the streaming site (success, error shape, stderr safety, spawn-site grep).
|
||||
- Integration: date parity + both fail-soft arms + the phase-179 `app.api.sync.file_commit_dates` monkeypatch point still exercised by `test_sync_git_freeze.py`.
|
||||
- Coverage: **>90%** on `app/`; `scripts/git_sync.py` fully covered by the unit family.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `stream_git_lines` exists with the stderr-thread + terminate-on-close shape and the `run_git`-shaped `GitSyncError`; the module docstring names both spawn sites and the grep pin enforces it.
|
||||
- [ ] `file_commit_dates` streams, returns byte-identical dicts for the parity fixture, and keeps both fail-soft arms and warnings.
|
||||
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` green.
|
||||
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 03 — E2E: the dates the owner reads come through the stream (the phase's dedicated Playwright suite)
|
||||
|
||||
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Pin the phase-106 D2 promise through the browser with the streaming walk: after syncing a real two-commit `file://` repo, the dates shown for each document are that file's LAST-COMMIT dates (not the sync date, not the clone date), uncommitted files fall back to mtime, and a manual date correction survives a re-sync. Then the full gates, the docs pass, the atomic commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_git_log_stream_dates.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_git_source_dates.py` + `test_document_dates.py` shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, `BOR_GIT_SOURCES=file://<fixture repo>`, the shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`, the `_git` helper with `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` set per commit so dates are exact and stable).
|
||||
- Fixture repo: commit A (dated 2026-01-05) adds `notes/old.md` + `notes/changed.md`; commit B (dated 2026-03-20) modifies `notes/changed.md` only; plus an untracked `notes/untracked.md` written directly into the CHECKOUT after cloning (or, if the sync re-clones, into the source dir the run walks — whichever the house fixture model uses; state which in the docstring).
|
||||
- **Test 1 `test_document_dates_are_the_last_commit_dates`:** sync from the Sources page → `notes/old.md`'s document date is 2026-01-05 and `notes/changed.md`'s is 2026-03-20 (assert via `GET /api/docs/{id}` through `page.request` and the document page's date surface), neither equal to the run's own timestamp.
|
||||
- **Test 2 `test_uncommitted_file_falls_back_to_mtime`:** the untracked file's document date equals its file mtime (the fail-soft contract through the streaming path) and the run reaches `success` with no error banner.
|
||||
- **Test 3 `test_manual_date_survives_a_resync` (the D1 lock):** set a document's date through the existing date-edit surface, re-sync, and the manual date is unchanged while the other documents refresh from the stream.
|
||||
- **Test 4 `test_resync_is_unchanged_and_dates_hold`:** a second sync with nothing changed reports all `unchanged` and identical dates (the stream re-parses, it does not drift).
|
||||
- Non-ASCII/quoted path behaviour: record what the walk does TODAY with a `core.quotepath`-quoted path in the module docstring and pin it as UNCHANGED behaviour (whatever the run shows — this phase changes transport, not parsing; if the behaviour is wrong, note it for a future phase rather than fixing it here).
|
||||
2. **Docs pass:** `README.md` + `AGENTS.md` in full, every inconsistency fixed (pre-existing included). `AGENTS.md` rule 11 gains the one-clause reference to `stream_git_lines` (both sites off-loop via `asyncio.to_thread`); `README.md` gains the ops note (the commit-date walk streams; the per-source date dict is still held for the run — phase 186 A3).
|
||||
3. **The full gates:**
|
||||
- `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` — green **in isolation**.
|
||||
- `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` and `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` — green in isolation (the date families this phase must not disturb).
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
|
||||
- `uv run ruff check . && uv run pyright` — clean.
|
||||
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `perf(sync): stream the git commit-date walk instead of buffering git log in RAM` — covering `scripts/git_sync.py`, the unit + integration + E2E files, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/186_git_log_streaming_dates/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the dedicated suite + the two date E2E families in isolation.
|
||||
- Coverage: **>90%** on `app/`; ruff + pyright clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` green **in isolation** (last-commit dates, mtime fallback, the D1 manual lock, the unchanged re-sync).
|
||||
- [ ] `tests/e2e/test_git_source_dates.py` and `tests/e2e/test_document_dates.py` green in isolation.
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
|
||||
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the rule-11 clause included).
|
||||
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
|
||||
Reference in New Issue
Block a user