phase: 186_git_log_streaming_dates
## Final report — phase 186 `git_log_streaming_dates`
**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.
### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |
Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.
### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.
Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.
⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).
Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
This commit is contained in:
@@ -183,7 +183,7 @@ still renders its own text.
|
||||
The document's meta row (the chat modal **and** the standalone viewer at
|
||||
`/document.html`) carries a **Copy path** button: one click copies the
|
||||
document's `source/path` (the exact form the `read` tool, the chips, and
|
||||
the Sources table use) to the clipboard; on a non-secure (http) homelab
|
||||
the RAG table use) to the clipboard; on a non-secure (http) homelab
|
||||
origin the browser rejects the clipboard, so the click instead offers the
|
||||
full path in an inline fallback field that selects itself on focus —
|
||||
click or Tab, then copy (phase 153).
|
||||
@@ -373,9 +373,9 @@ the same page (permanent; admin-only). Admin-only flow page.
|
||||
## Admin & Sign-in
|
||||
|
||||
Brain of Reese has exactly **one account: the admin (you)**. Signing in
|
||||
unlocks the **full Sources catalog**, the **answer-tuning** controls, and
|
||||
**token management**. **Shared chats** are the only content that stays open
|
||||
to anonymous visitors.
|
||||
unlocks the **full RAG catalog** (`/sources.html`), the **answer-tuning**
|
||||
controls, and **token management**. **Shared chats** are the only content
|
||||
that stays open to anonymous visitors.
|
||||
|
||||
### Setup (one-time)
|
||||
|
||||
@@ -415,8 +415,8 @@ The admin can hand out access without sharing the admin password.
|
||||
caches it in `localStorage["bor.token"]` for silent re-auth on reloads.
|
||||
3. **Revoke:** click **Revoke** on the token's row. Revocation is immediate.
|
||||
|
||||
A token user can chat and open documents — nothing else. The Sources
|
||||
catalog, git sources, tuning, and history stay admin-only.
|
||||
A token user can chat and open documents — nothing else. The RAG catalog,
|
||||
git sources, tuning, and history stay admin-only.
|
||||
|
||||
---
|
||||
|
||||
@@ -679,7 +679,8 @@ only) runs the whole source refresh in one click:
|
||||
`tests/integration/test_import_session_retention.py` (the identity-map
|
||||
bound, the no-`content` prune `SELECT` with no per-row lazy load, the
|
||||
3-commits-vs-1 cadence, both prune guards on light rows) and end to end
|
||||
through the Sources page by `tests/e2e/test_sync_session_reset.py` — two
|
||||
through the RAG page (`/sources.html`) by
|
||||
`tests/e2e/test_sync_session_reset.py` — two
|
||||
module apps on their own ports (`E2E_APP_PORT_SYNCRESET` /
|
||||
`E2E_APP_PORT_SYNCRESET_ZERO`), one per cadence: the ten-file walk lands
|
||||
every document exactly once with the totals the stat cards show, the
|
||||
@@ -710,10 +711,54 @@ only) runs the whole source refresh in one click:
|
||||
the only deferred column, all four loads share the helper, no
|
||||
`raiseload`/`noload` anywhere in the importer) and, through the UI,
|
||||
`tests/e2e/test_fast_path_deferred_content.py` (a hand-nulled summary
|
||||
backfills to its byte-identical text from the Sources-page re-sync with
|
||||
backfills to its byte-identical text from the RAG-page
|
||||
(`/sources.html`) re-sync with
|
||||
exactly one `is_summary` chunk, an unchanged doc's viewer still shows
|
||||
the full stored text, and a rewritten file re-indexes to the new text
|
||||
with the next sync all-`unchanged` again).
|
||||
- **The commit-date walk streams** (phase 186). A git source's per-file
|
||||
last-commit dates come from one `git log --name-only --format=@@%cI`
|
||||
over the checkout — a walk that scales with the repository's **entire
|
||||
history**, not its working tree (the incident box measured 346,829
|
||||
lines / 15 MB for one source, and the removed `server`/nextcloud
|
||||
source showed how little that has to do with the files on disk). It no
|
||||
longer arrives as one string: `stream_git_lines` (the second,
|
||||
documented spawn site beside `run_git`) yields it line by line off the
|
||||
pipe, so no `git log` is ever held in RAM, and stderr is drained on a
|
||||
worker thread so a chatty git cannot deadlock the pipe. The parser
|
||||
takes any iterable of lines (`_parse_commit_dates(str | Iterable[str])`
|
||||
— a string still behaves exactly as before), so the grammar is
|
||||
untouched. The dates are byte-identical — first-sighting-wins, the same
|
||||
fail-soft arms (`{}` + the same warning lines + the mtime fallback),
|
||||
the D1 manual date untouched — and what IS still retained is the
|
||||
per-source date **dict**, held by the run for its whole duration
|
||||
(phase 186 A3: it is ~10 % of the cost this removed, and bounding it
|
||||
would change dates, not just memory). Pinned by
|
||||
`tests/unit/test_git_sync.py` (the stream, the `run_git`-shaped
|
||||
`GitSyncError`, the 340 KB-stderr no-deadlock case, the
|
||||
terminate-on-abandon case, and `test_only_two_git_spawn_sites`, the AST
|
||||
pin that enumerates EVERY `subprocess.*` spawn in `app/` and
|
||||
`scripts/` — the two git sites plus the one non-git exception, the
|
||||
boot-time Valkey probe in `app/core/caching.py` — so a new spawn
|
||||
anywhere fails the suite), `tests/integration/test_git_file_dates.py`
|
||||
(the streamed dict EQUALS the buffered one across a commit-1-only
|
||||
file, a commit-2-modified file, a subdirectory file and a
|
||||
never-committed file that falls back to its mtime) and, through the
|
||||
browser, `tests/e2e/test_git_log_stream_dates.py` (a real two-commit
|
||||
`file://` source synced from the RAG page (`/sources.html`) shows each
|
||||
file's own last-commit date in the table and the viewer, an
|
||||
uncommitted file its mtime, the owner's manual date survives the
|
||||
re-sync, and an unchanged re-sync moves no date at all). One
|
||||
pre-existing quirk is recorded while we are here, **unchanged** and
|
||||
pinned by that suite: with git's default `core.quotepath=true` a
|
||||
non-ASCII path reaches the walk quoted and octal-escaped
|
||||
(`"notes/caf\303\251.md"`), the parser's whitespace split and
|
||||
`\`→`/` normalisation turn it into a key that can never match the
|
||||
importer's path, and such a document therefore shows its **file
|
||||
mtime** — the day the checkout last wrote the file — instead of its
|
||||
commit date. Fixing it means changing the quoting grammar — a date
|
||||
change for every non-ASCII document already in a KB, so it is a future
|
||||
phase's call, not a transport phase's.
|
||||
- **Cancel**: while a run is in flight, a **Cancel** button appears
|
||||
next to the sync button; clicking it posts `POST /api/sync/cancel`
|
||||
(admin only) and registers a cooperative cancel (`202`) — the run
|
||||
@@ -960,7 +1005,7 @@ first-class document:
|
||||
`BOR_IMAGE_DIR/<uuid>.<ext>` and served from that copy (uploads are
|
||||
replaced and checkouts re-cloned, so the served image outlives its
|
||||
source file).
|
||||
- **Retrieved & displayed.** Image docs appear in the Sources catalog
|
||||
- **Retrieved & displayed.** Image docs appear in the RAG catalog
|
||||
with a small lazy-loaded thumbnail (progressive enhancement — a fetch
|
||||
failure falls back to the doc icon), the document viewer renders the
|
||||
image with the description below it as **wrapping prose** (phase 165 —
|
||||
@@ -1011,7 +1056,7 @@ re-import by hand. In the viewer the graphic **spans the image card**: a
|
||||
carries no intrinsic size, so `doc.format === "svg"` adds
|
||||
`.doc-image-img--vector` (`width: 100%` + `object-fit: contain`) instead
|
||||
of letting the drawing sit in the browser's 300×150 default box; the
|
||||
chat figure and the Sources thumbnail keep their fixed caps (96px /
|
||||
chat figure and the RAG-table thumbnail keep their fixed caps (96px /
|
||||
48px) and the modal is the full-size preview. Question images (the
|
||||
paperclip) stay raster-only: an `.svg` upload is a 422.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user