phase: 187_sync_peak_memory_line
**Phase 187 — final verification pass: all criteria met, repo 100% green.**
Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.
Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)
Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.
Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).
Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.
Next pending phase: `188_chunks_hnsw_index`.
This commit is contained in:
@@ -622,7 +622,9 @@ only) runs the whole source refresh in one click:
|
||||
the sync area (`role="status"`, polite — phase 150 folded the
|
||||
pre-phase result-line announcer into the box). At settle the box
|
||||
STAYS visible to carry the result line — the counts line
|
||||
("N added · …") after a success, "Sync cancelled" after a cancel —
|
||||
("N added · …") after a success, "Sync cancelled" after a cancel, each
|
||||
trailed by the run's peak memory (" · peak 1936 MB", phase 187; nothing
|
||||
at all when the reading is null) —
|
||||
and the button leaves the running state (the "Synced <stamp>" pill
|
||||
after a success — the date-qualified freshness stamp (phase 154):
|
||||
`HH:MM` on the same local day, `M/D, h:MM AM/PM` otherwise
|
||||
@@ -759,6 +761,32 @@ only) runs the whole source refresh in one click:
|
||||
commit date. Fixing it means changing the quoting grammar — a date
|
||||
change for every non-ASCII document already in a KB, so it is a future
|
||||
phase's call, not a transport phase's.
|
||||
- **The run's memory is a logged number and a line on the page**
|
||||
(phase 187). Every sync lifecycle line ends with `rss_mb=` — and every
|
||||
terminal with `rss_mb= rss_peak_mb=` — read through
|
||||
`app/core/memory.py` (PROCESS RSS, never the host's cgroup figure,
|
||||
which counts page cache; an unknowable reading prints `?`, never
|
||||
`None`), and the SAME pair rides `GET /api/sync/status` as the
|
||||
additive `rss_mb` / `rss_peak_mb` keys (`int | null`). The RAG page
|
||||
(`/sources.html`) appends it to the result line it already spoke —
|
||||
`3 added · 1 unchanged · peak 1936 MB`, `Sync cancelled · peak 1936 MB`
|
||||
— no new control, and nothing at all when the pair is null. "How much
|
||||
RAM did that sync use?" is therefore one grep of the app's own log
|
||||
**or** one
|
||||
glance at the page that started the run, instead of a dig through
|
||||
`journalctl` unit-teardown lines from three boots (the 2026-10-15
|
||||
incident). The fields are appended, so every older grep still matches,
|
||||
and the sampler keeps the pair honest (`peak >= current` — a kernel
|
||||
high-water refreshed lazily is raised to the current reading, because a
|
||||
peak below the run's own footprint is a false number, not a smaller one).
|
||||
Pinned by `tests/unit/test_core_memory.py` (every helper branch, the
|
||||
never-raise contract), `tests/unit/test_sync_button.py` (the sampling
|
||||
points, the 5 s throttle, the pair's invariant),
|
||||
`tests/integration/test_sync_api.py` (the four lifecycle lines, the
|
||||
additive keys, the terminals keeping the final pair, the anonymous 403)
|
||||
and end to end in the browser by
|
||||
`tests/e2e/test_sync_peak_memory_visible.py`. See *Operational notes:
|
||||
the sync's memory (phase 187)*.
|
||||
- **Cancel**: while a run is in flight, a **Cancel** button appears
|
||||
next to the sync button; clicking it posts `POST /api/sync/cancel`
|
||||
(admin only) and registers a cooperative cancel (`202`) — the run
|
||||
@@ -1516,6 +1544,87 @@ under `.agents/phases/`.
|
||||
> overriding it) is correct; unsetting it would hand every client
|
||||
> uncompressed JSON.
|
||||
|
||||
### Operational notes: the sync's memory (phase 187)
|
||||
|
||||
The 2026-10-15 incident made "how much RAM did that sync use?" a dig
|
||||
through `journalctl` unit-teardown lines from three boots — and the boot
|
||||
that killed the box had no teardown record at all, because it never shut
|
||||
down cleanly. Every sync lifecycle line now carries the number, so the
|
||||
question is one grep:
|
||||
|
||||
```
|
||||
2026-10-15 04:12:07 INFO app.api.sync :: sync: done detail={'files': 27411, …} rss_mb=412 rss_peak_mb=1936
|
||||
```
|
||||
|
||||
```bash
|
||||
grep ' :: ' server.log | grep ' rss_peak_mb=' # every sync terminal
|
||||
grep ' :: ' server.log | grep 'sync: started ' # where each run began
|
||||
grep ' :: ' server.log | grep 'sync: failed state=' # the failed ones, with the error
|
||||
```
|
||||
|
||||
- **`rss_mb=`** is the process's CURRENT resident set size in MB (Linux:
|
||||
`VmRSS` from `/proc/self/status`), read at the moment that line is
|
||||
written — the run's start, the import's summary line, and every
|
||||
terminal.
|
||||
- **`rss_peak_mb=`** is the process HIGH-WATER mark
|
||||
(`getrusage(RUSAGE_SELF).ru_maxrss`): the most the process ever held,
|
||||
which is the number the incident was actually about. It appears on the
|
||||
three terminals (`sync: done …`, `sync: cancelled by user …`,
|
||||
`sync: failed state=failed …`) beside the final counts. The two numbers
|
||||
come from different kernel counters and both floor to whole MB, and the
|
||||
kernel refreshes the high-water **lazily**, so a raw `rss_peak_mb()` can
|
||||
sit under a raw `rss_mb()` (measured: 164 against 162). The run's own
|
||||
sampling point (`app.api.sync._sample_rss`) therefore lifts a lagging
|
||||
peak to the current reading before anything publishes it — so on every
|
||||
sync line, on `/api/sync/status` and on the page you always get
|
||||
`peak >= current`. An unknowable peak (`?` / `null`) is never fabricated
|
||||
from the current reading.
|
||||
- **`?` means "unknowable here", never 0 and never `None`** —
|
||||
`app/core/memory.py` never raises, so a platform without `/proc` (or
|
||||
without `resource`) degrades the reading and the sync keeps running.
|
||||
- **Process vs cgroup — know which number you are reading.** These are
|
||||
PROCESS numbers. The host's `/sys/fs/cgroup/memory.peak` — and the
|
||||
`systemd` "… memory peak" line the incident started from — is a
|
||||
different, larger figure, because a cgroup counts **page cache** too:
|
||||
that box's app cgroup showed ~2.16 GB of `file` against ~211 MB of
|
||||
`anon` while idle. Neither figure is wrong; the log says which one it
|
||||
is (LOCKED A1), and `app.core.memory.cgroup_peak_mb()` exists for an
|
||||
operator who wants the other one for context.
|
||||
- **Append-only** (LOCKED A2): the fields trail the existing ones on the
|
||||
pre-phase lines (`sync: started repos=… origin=… git=… local=…`,
|
||||
`sync: done detail=…`, `sync: cancelled by user state=cancelled
|
||||
files_done=… files_total=…`, `import: summary files=… … formats=…`),
|
||||
and the failed funnel keeps its `logger.exception` traceback while
|
||||
gaining ONE sanitized one-line summary (`error=` carries exactly the
|
||||
masked string the status field does). So every grep written before this
|
||||
phase still matches.
|
||||
|
||||
- **On the page, not only in the log.** The same pair rides
|
||||
`GET /api/sync/status` (admin-only, like every sync surface) as the
|
||||
additive `rss_mb` / `rss_peak_mb` keys, and the RAG page
|
||||
(`/sources.html`) appends them to the result line it already speaks —
|
||||
`3 added · 1 unchanged · peak 1936 MB`, or
|
||||
`Sync cancelled · peak 1936 MB`. A terminal KEEPS
|
||||
the final reading next to the final counts, so a page reloaded an hour
|
||||
later still shows what that run reached, and `sync: done …` logs the
|
||||
very same numbers (one reading, two surfaces — the page and the log
|
||||
can never disagree).
|
||||
- **Sampled, never per-file** (LOCKED A4): the run reads its own memory
|
||||
at the run start, at each phase transition (`import` → `overview` →
|
||||
`summaries`), in every terminal, and — throttled to at most one read
|
||||
per 5 s — from the import's per-file progress hook, so the number
|
||||
moves while you watch a long walk without costing 27k `/proc` reads.
|
||||
- **`null` renders NOTHING.** No reading (idle, a run that predates this
|
||||
phase, the upload job's status, a platform that cannot know) shows no
|
||||
clause at all — never `peak ? MB`, never `peak 0 MB`. The failed
|
||||
terminal's box stays empty by the existing contract (the banner and
|
||||
the modal carry the error, byte-identical); a failed run's numbers are
|
||||
in its log line and in the status payload.
|
||||
|
||||
There is deliberately **no** memory audit script and **no** metrics
|
||||
endpoint (LOCKED A3/A2 — no aux services, A12): the grep above is the
|
||||
interface.
|
||||
|
||||
---
|
||||
|
||||
## Configuration reference
|
||||
|
||||
Reference in New Issue
Block a user