phase: 186_git_log_streaming_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m42s
Build and Push Containers / build-and-push-db (push) Successful in 14s

## Final report — phase 186 `git_log_streaming_dates`

**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.

### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |

Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.

### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.

Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.

⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).

Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
This commit is contained in:
2026-10-16 07:50:16 -04:00
parent 632fb88e43
commit 644f7de696
36 changed files with 2671 additions and 48 deletions
+55 -10
View File
@@ -183,7 +183,7 @@ still renders its own text.
The document's meta row (the chat modal **and** the standalone viewer at
`/document.html`) carries a **Copy path** button: one click copies the
document's `source/path` (the exact form the `read` tool, the chips, and
the Sources table use) to the clipboard; on a non-secure (http) homelab
the RAG table use) to the clipboard; on a non-secure (http) homelab
origin the browser rejects the clipboard, so the click instead offers the
full path in an inline fallback field that selects itself on focus —
click or Tab, then copy (phase 153).
@@ -373,9 +373,9 @@ the same page (permanent; admin-only). Admin-only flow page.
## Admin & Sign-in
Brain of Reese has exactly **one account: the admin (you)**. Signing in
unlocks the **full Sources catalog**, the **answer-tuning** controls, and
**token management**. **Shared chats** are the only content that stays open
to anonymous visitors.
unlocks the **full RAG catalog** (`/sources.html`), the **answer-tuning**
controls, and **token management**. **Shared chats** are the only content
that stays open to anonymous visitors.
### Setup (one-time)
@@ -415,8 +415,8 @@ The admin can hand out access without sharing the admin password.
caches it in `localStorage["bor.token"]` for silent re-auth on reloads.
3. **Revoke:** click **Revoke** on the token's row. Revocation is immediate.
A token user can chat and open documents — nothing else. The Sources
catalog, git sources, tuning, and history stay admin-only.
A token user can chat and open documents — nothing else. The RAG catalog,
git sources, tuning, and history stay admin-only.
---
@@ -679,7 +679,8 @@ only) runs the whole source refresh in one click:
`tests/integration/test_import_session_retention.py` (the identity-map
bound, the no-`content` prune `SELECT` with no per-row lazy load, the
3-commits-vs-1 cadence, both prune guards on light rows) and end to end
through the Sources page by `tests/e2e/test_sync_session_reset.py` — two
through the RAG page (`/sources.html`) by
`tests/e2e/test_sync_session_reset.py` — two
module apps on their own ports (`E2E_APP_PORT_SYNCRESET` /
`E2E_APP_PORT_SYNCRESET_ZERO`), one per cadence: the ten-file walk lands
every document exactly once with the totals the stat cards show, the
@@ -710,10 +711,54 @@ only) runs the whole source refresh in one click:
the only deferred column, all four loads share the helper, no
`raiseload`/`noload` anywhere in the importer) and, through the UI,
`tests/e2e/test_fast_path_deferred_content.py` (a hand-nulled summary
backfills to its byte-identical text from the Sources-page re-sync with
backfills to its byte-identical text from the RAG-page
(`/sources.html`) re-sync with
exactly one `is_summary` chunk, an unchanged doc's viewer still shows
the full stored text, and a rewritten file re-indexes to the new text
with the next sync all-`unchanged` again).
- **The commit-date walk streams** (phase 186). A git source's per-file
last-commit dates come from one `git log --name-only --format=@@%cI`
over the checkout — a walk that scales with the repository's **entire
history**, not its working tree (the incident box measured 346,829
lines / 15 MB for one source, and the removed `server`/nextcloud
source showed how little that has to do with the files on disk). It no
longer arrives as one string: `stream_git_lines` (the second,
documented spawn site beside `run_git`) yields it line by line off the
pipe, so no `git log` is ever held in RAM, and stderr is drained on a
worker thread so a chatty git cannot deadlock the pipe. The parser
takes any iterable of lines (`_parse_commit_dates(str | Iterable[str])`
— a string still behaves exactly as before), so the grammar is
untouched. The dates are byte-identical — first-sighting-wins, the same
fail-soft arms (`{}` + the same warning lines + the mtime fallback),
the D1 manual date untouched — and what IS still retained is the
per-source date **dict**, held by the run for its whole duration
(phase 186 A3: it is ~10 % of the cost this removed, and bounding it
would change dates, not just memory). Pinned by
`tests/unit/test_git_sync.py` (the stream, the `run_git`-shaped
`GitSyncError`, the 340 KB-stderr no-deadlock case, the
terminate-on-abandon case, and `test_only_two_git_spawn_sites`, the AST
pin that enumerates EVERY `subprocess.*` spawn in `app/` and
`scripts/` — the two git sites plus the one non-git exception, the
boot-time Valkey probe in `app/core/caching.py` — so a new spawn
anywhere fails the suite), `tests/integration/test_git_file_dates.py`
(the streamed dict EQUALS the buffered one across a commit-1-only
file, a commit-2-modified file, a subdirectory file and a
never-committed file that falls back to its mtime) and, through the
browser, `tests/e2e/test_git_log_stream_dates.py` (a real two-commit
`file://` source synced from the RAG page (`/sources.html`) shows each
file's own last-commit date in the table and the viewer, an
uncommitted file its mtime, the owner's manual date survives the
re-sync, and an unchanged re-sync moves no date at all). One
pre-existing quirk is recorded while we are here, **unchanged** and
pinned by that suite: with git's default `core.quotepath=true` a
non-ASCII path reaches the walk quoted and octal-escaped
(`"notes/caf\303\251.md"`), the parser's whitespace split and
`\`→`/` normalisation turn it into a key that can never match the
importer's path, and such a document therefore shows its **file
mtime** — the day the checkout last wrote the file — instead of its
commit date. Fixing it means changing the quoting grammar — a date
change for every non-ASCII document already in a KB, so it is a future
phase's call, not a transport phase's.
- **Cancel**: while a run is in flight, a **Cancel** button appears
next to the sync button; clicking it posts `POST /api/sync/cancel`
(admin only) and registers a cooperative cancel (`202`) — the run
@@ -960,7 +1005,7 @@ first-class document:
`BOR_IMAGE_DIR/<uuid>.<ext>` and served from that copy (uploads are
replaced and checkouts re-cloned, so the served image outlives its
source file).
- **Retrieved & displayed.** Image docs appear in the Sources catalog
- **Retrieved & displayed.** Image docs appear in the RAG catalog
with a small lazy-loaded thumbnail (progressive enhancement — a fetch
failure falls back to the doc icon), the document viewer renders the
image with the description below it as **wrapping prose** (phase 165 —
@@ -1011,7 +1056,7 @@ re-import by hand. In the viewer the graphic **spans the image card**: a
carries no intrinsic size, so `doc.format === "svg"` adds
`.doc-image-img--vector` (`width: 100%` + `object-fit: contain`) instead
of letting the drawing sit in the browser's 300×150 default box; the
chat figure and the Sources thumbnail keep their fixed caps (96px /
chat figure and the RAG-table thumbnail keep their fixed caps (96px /
48px) and the modal is the full-size preview. Question images (the
paperclip) stay raster-only: an `.svg` upload is a 422.