phase: 187_sync_peak_memory_line
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 187 — final verification pass: all criteria met, repo 100% green.**

Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.

Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)

Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.

Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).

Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.

Next pending phase: `188_chunks_hnsw_index`.
This commit is contained in:
2026-10-16 15:22:51 -04:00
parent 644f7de696
commit c054b47df5
72 changed files with 3826 additions and 96 deletions
+110 -1
View File
@@ -622,7 +622,9 @@ only) runs the whole source refresh in one click:
the sync area (`role="status"`, polite — phase 150 folded the
pre-phase result-line announcer into the box). At settle the box
STAYS visible to carry the result line — the counts line
("N added · …") after a success, "Sync cancelled" after a cancel —
("N added · …") after a success, "Sync cancelled" after a cancel, each
trailed by the run's peak memory (" · peak 1936 MB", phase 187; nothing
at all when the reading is null) —
and the button leaves the running state (the "Synced <stamp>" pill
after a success — the date-qualified freshness stamp (phase 154):
`HH:MM` on the same local day, `M/D, h:MM AM/PM` otherwise
@@ -759,6 +761,32 @@ only) runs the whole source refresh in one click:
commit date. Fixing it means changing the quoting grammar — a date
change for every non-ASCII document already in a KB, so it is a future
phase's call, not a transport phase's.
- **The run's memory is a logged number and a line on the page**
(phase 187). Every sync lifecycle line ends with `rss_mb=` — and every
terminal with `rss_mb= rss_peak_mb=` — read through
`app/core/memory.py` (PROCESS RSS, never the host's cgroup figure,
which counts page cache; an unknowable reading prints `?`, never
`None`), and the SAME pair rides `GET /api/sync/status` as the
additive `rss_mb` / `rss_peak_mb` keys (`int | null`). The RAG page
(`/sources.html`) appends it to the result line it already spoke —
`3 added · 1 unchanged · peak 1936 MB`, `Sync cancelled · peak 1936 MB`
— no new control, and nothing at all when the pair is null. "How much
RAM did that sync use?" is therefore one grep of the app's own log
**or** one
glance at the page that started the run, instead of a dig through
`journalctl` unit-teardown lines from three boots (the 2026-10-15
incident). The fields are appended, so every older grep still matches,
and the sampler keeps the pair honest (`peak >= current` — a kernel
high-water refreshed lazily is raised to the current reading, because a
peak below the run's own footprint is a false number, not a smaller one).
Pinned by `tests/unit/test_core_memory.py` (every helper branch, the
never-raise contract), `tests/unit/test_sync_button.py` (the sampling
points, the 5 s throttle, the pair's invariant),
`tests/integration/test_sync_api.py` (the four lifecycle lines, the
additive keys, the terminals keeping the final pair, the anonymous 403)
and end to end in the browser by
`tests/e2e/test_sync_peak_memory_visible.py`. See *Operational notes:
the sync's memory (phase 187)*.
- **Cancel**: while a run is in flight, a **Cancel** button appears
next to the sync button; clicking it posts `POST /api/sync/cancel`
(admin only) and registers a cooperative cancel (`202`) — the run
@@ -1516,6 +1544,87 @@ under `.agents/phases/`.
> overriding it) is correct; unsetting it would hand every client
> uncompressed JSON.
### Operational notes: the sync's memory (phase 187)
The 2026-10-15 incident made "how much RAM did that sync use?" a dig
through `journalctl` unit-teardown lines from three boots — and the boot
that killed the box had no teardown record at all, because it never shut
down cleanly. Every sync lifecycle line now carries the number, so the
question is one grep:
```
2026-10-15 04:12:07 INFO app.api.sync :: sync: done detail={'files': 27411, …} rss_mb=412 rss_peak_mb=1936
```
```bash
grep ' :: ' server.log | grep ' rss_peak_mb=' # every sync terminal
grep ' :: ' server.log | grep 'sync: started ' # where each run began
grep ' :: ' server.log | grep 'sync: failed state=' # the failed ones, with the error
```
- **`rss_mb=`** is the process's CURRENT resident set size in MB (Linux:
`VmRSS` from `/proc/self/status`), read at the moment that line is
written — the run's start, the import's summary line, and every
terminal.
- **`rss_peak_mb=`** is the process HIGH-WATER mark
(`getrusage(RUSAGE_SELF).ru_maxrss`): the most the process ever held,
which is the number the incident was actually about. It appears on the
three terminals (`sync: done …`, `sync: cancelled by user …`,
`sync: failed state=failed …`) beside the final counts. The two numbers
come from different kernel counters and both floor to whole MB, and the
kernel refreshes the high-water **lazily**, so a raw `rss_peak_mb()` can
sit under a raw `rss_mb()` (measured: 164 against 162). The run's own
sampling point (`app.api.sync._sample_rss`) therefore lifts a lagging
peak to the current reading before anything publishes it — so on every
sync line, on `/api/sync/status` and on the page you always get
`peak >= current`. An unknowable peak (`?` / `null`) is never fabricated
from the current reading.
- **`?` means "unknowable here", never 0 and never `None`** —
`app/core/memory.py` never raises, so a platform without `/proc` (or
without `resource`) degrades the reading and the sync keeps running.
- **Process vs cgroup — know which number you are reading.** These are
PROCESS numbers. The host's `/sys/fs/cgroup/memory.peak` — and the
`systemd` "… memory peak" line the incident started from — is a
different, larger figure, because a cgroup counts **page cache** too:
that box's app cgroup showed ~2.16 GB of `file` against ~211 MB of
`anon` while idle. Neither figure is wrong; the log says which one it
is (LOCKED A1), and `app.core.memory.cgroup_peak_mb()` exists for an
operator who wants the other one for context.
- **Append-only** (LOCKED A2): the fields trail the existing ones on the
pre-phase lines (`sync: started repos=… origin=… git=… local=…`,
`sync: done detail=…`, `sync: cancelled by user state=cancelled
files_done=… files_total=…`, `import: summary files=… … formats=…`),
and the failed funnel keeps its `logger.exception` traceback while
gaining ONE sanitized one-line summary (`error=` carries exactly the
masked string the status field does). So every grep written before this
phase still matches.
- **On the page, not only in the log.** The same pair rides
`GET /api/sync/status` (admin-only, like every sync surface) as the
additive `rss_mb` / `rss_peak_mb` keys, and the RAG page
(`/sources.html`) appends them to the result line it already speaks —
`3 added · 1 unchanged · peak 1936 MB`, or
`Sync cancelled · peak 1936 MB`. A terminal KEEPS
the final reading next to the final counts, so a page reloaded an hour
later still shows what that run reached, and `sync: done …` logs the
very same numbers (one reading, two surfaces — the page and the log
can never disagree).
- **Sampled, never per-file** (LOCKED A4): the run reads its own memory
at the run start, at each phase transition (`import` → `overview` →
`summaries`), in every terminal, and — throttled to at most one read
per 5 s — from the import's per-file progress hook, so the number
moves while you watch a long walk without costing 27k `/proc` reads.
- **`null` renders NOTHING.** No reading (idle, a run that predates this
phase, the upload job's status, a platform that cannot know) shows no
clause at all — never `peak ? MB`, never `peak 0 MB`. The failed
terminal's box stays empty by the existing contract (the banner and
the modal carry the error, byte-identical); a failed run's numbers are
in its log line and in the status payload.
There is deliberately **no** memory audit script and **no** metrics
endpoint (LOCKED A3/A2 — no aux services, A12): the grep above is the
interface.
---
## Configuration reference