ducoterra
a45f744604
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
1295 lines
70 KiB
Python
1295 lines
70 KiB
Python
"""Sources sync API — one-click KB mirror (phase 32, task 01).
|
|
|
|
Admin-only ``POST /api/sync`` + ``GET /api/sync/status`` behind the
|
|
existing :func:`app.core.auth.require_admin` (A10 extended, phase 16
|
|
pattern — the public API surface stays stateless, the signed cookie
|
|
remains the only session state, same as ``/api/steering``).
|
|
|
|
The button's backend runs the full document sync **in-process** (A12
|
|
untouched — no queue, no new services): one ``asyncio`` background task
|
|
plus a module-level :class:`SyncStatus` that the UI polls every 2 s
|
|
(task 02). One sync at a time — ``POST`` while a run is in flight is
|
|
409; the status object is authoritative, so the UI can never sit on a
|
|
stale button state (§7.4 adaptation, phase locked decisions). Phase
|
|
134 (task 01) adds the cancel: ``POST /api/sync/cancel`` (admin-only,
|
|
same router) registers a COOPERATIVE cancel while a run is in flight
|
|
(202, 409 ``no sync is running`` otherwise), and the run lands in the
|
|
new ``cancelled`` TERMINAL state — a cancel is intent, not failure
|
|
(``error = None``, ``detail = {}``, the final counts kept; the cancel
|
|
paragraph below).
|
|
|
|
Pipeline (the canonical "mirror the sources" action — phase locked
|
|
decisions):
|
|
|
|
1. verify ``embed`` + summary model availability — fail fast before
|
|
any clone (:func:`app.rag.llm.check_models`, phase 41): a dead
|
|
model endpoint aborts the run naming the unavailable model, before
|
|
source resolution or any ``clone_or_pull``;
|
|
2. resolve the effective sources — the ``git_sources`` DB rows (git
|
|
**and** local, phase 38), else the ``BOR_GIT_SOURCES`` fallback
|
|
(git-only)
|
|
(:func:`app.rag.git_sources.effective_sources`, shared with the
|
|
CLI) — empty on both origins (no git rows, no local rows, no env
|
|
URLs) fails loudly (``no sources configured (git or local)``)
|
|
instead of silently importing the legacy local directories;
|
|
3. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull`
|
|
with the phase-121 clone URL (:func:`app.rag.git_sources.clone_url_for`
|
|
— the row's ``token`` column injected as
|
|
``https://x-access-token:<token>@…`` only for https? rows; NULL
|
|
token → the bare stored URL verbatim, so public repos and legacy
|
|
embedded-token rows clone exactly as before) and the row's
|
|
BRANCH PIN as the third argument (phase 135 — a fresh clone checks
|
|
it out, ``git clone --branch``; an existing checkout is guarded — a
|
|
mismatched branch fails loudly, the remove-and-re-add error — and
|
|
tracked, ``git fetch origin <b>`` + ``git merge --ff-only
|
|
FETCH_HEAD``; ``branch=None``, i.e. every pre-phase-135 row and
|
|
every env-fallback row, keeps the pre-phase argv byte-identical)
|
|
into
|
|
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
|
re-implemented; the checkout name stays on the bare URL —
|
|
credential-free); ``kind=local`` → the stored directory, re-verified
|
|
``.is_dir()`` **at sync time** (it may have moved/deleted since
|
|
add-time) — a missing directory raises ``local source missing:
|
|
<path>``; a failing clone or a missing local dir aborts before any
|
|
import;
|
|
4. ``import_sources(..., prune=True)`` over the single combined list
|
|
(git checkouts + local dirs), honoring each row's ``ignore_paths``
|
|
(phase 89 — the per-root ignore map is built in the same per-row
|
|
loop as the source list), its ``include_hidden`` flag (phase 105 —
|
|
the per-root hidden-folders map, same per-row construction), and
|
|
feeding each git checkout's per-file last-commit dates (phase 106,
|
|
D2 — the per-root date map, ``file_commit_dates`` after the clone,
|
|
same per-row construction; local rows contribute nothing and take
|
|
the importer's mtime fallback) — prune so files deleted upstream,
|
|
out of
|
|
a local dir, or newly matching an ignore pattern leave the index
|
|
(pruning covers the union; the CLI's no-prune default is unchanged);
|
|
5. when the import changed the KB (added + updated > 0),
|
|
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
|
(phase 31 trigger, best-effort inside) — and ``generate_
|
|
folder_summaries`` (phase 94, task 02) regenerates the stored folder
|
|
summaries (the drill-down ``ls``'s per-level descriptions): its gate
|
|
is the same change trigger — INCREMENTAL since phase 127 (task 03):
|
|
the run's changed documents (``ImportSummary.changed_paths``, the
|
|
import's added/updated/pruned ``(source, path)`` set) are passed as
|
|
``changed_docs=`` and ONLY their folder chains (each document's
|
|
folder up to its source root — ``affected_folder_keys``) are
|
|
rewritten, so a one-file change burns a handful of rows, not the
|
|
whole table (the ``scripts/import_docs.py`` CLI keeps the
|
|
byte-identical full regeneration — it passes no change set) **plus**,
|
|
on an unchanged re-sync, a GAP probe — a candidate folder (at least
|
|
1 doc) with no stored row (``missing_folder_summaries``, phase 96,
|
|
task 03): a gap fills ONLY the missing rows (``only_missing=True``;
|
|
the old table-empty first-sync trigger is subsumed exactly — an
|
|
empty table leaves every candidate missing), a complete table burns
|
|
zero ``lite`` calls. It is per-folder fail-soft (a ``lite`` outage
|
|
keeps the failed folders' previous rows — or leaves the row absent
|
|
— and never flips the run to ``failed``) and only flushes — this
|
|
run's own short-lived session commits (the phase-53 convention), so
|
|
a folder failure never blocks step 6's bump;
|
|
6. when the import changed the KB (added + updated + pruned > 0 — the
|
|
saved-chat invalidation gate, phase 53 task 02: a pruned document
|
|
can invalidate a saved answer that cited it, deliberately broader
|
|
than step 5's overview gate), the single-row ``sources_meta``
|
|
version counter is bumped exactly once in a short-lived session and
|
|
the resulting generation lands in the status detail as
|
|
``sources_version`` (an unchanged re-sync reports the current
|
|
generation without advancing it). A FAILED sync never bumps — the
|
|
run aborts in the ``failed`` state before this step.
|
|
|
|
Status is in memory: a restart mid-sync loses the running state
|
|
(accepted — the next click re-syncs idempotently). The status also
|
|
carries the phase-64 per-file progress — ``current_file`` (the
|
|
``source/relative/path`` the import is processing right now) plus
|
|
``files_done`` / ``files_total`` — null/0/0 before the import starts
|
|
(clone/pull reports no file yet) and in terminal states, which clear
|
|
``current_file`` but keep the run's final counts. What the pair COUNTS
|
|
changed in phase 194: on a progress caller (the UI sync is the only
|
|
one) it is the position in the pre-scan's PREDICTED change set —
|
|
``files_total`` the number of files that will actually be re-indexed,
|
|
``files_done`` the 1-based position in that set — which is what makes
|
|
the Sources page's ``N remaining`` say "10 remaining" on a ~10,000-file
|
|
KB with 10 changes (the owner report of 2026-10-19) instead of counting
|
|
the walk. ``BOR_SYNC_PENDING_COUNT=0`` restores the phase-193 walk
|
|
position byte-for-byte; the terminal ``detail`` counts stay the exact
|
|
surface either way.
|
|
|
|
Phase 134 (task 01) makes the machine FIVE-state: ``cancelled`` joins
|
|
``success`` / ``failed`` as a TERMINAL state — it carries
|
|
``finished_at``, clears ``current_file`` / ``phase`` /
|
|
``current_summary`` and the chunk position, and keeps the run's final
|
|
``files_done`` / ``files_total`` + summary counts (the phase-64/98
|
|
keep-final-counts convention) — but with ``error = None`` and
|
|
``detail = {}``: a cancel is intent, not failure, so the UI never
|
|
treats it as an error (no banner, no modal — a neutral result line).
|
|
``GET /api/sync/status`` gains nothing structurally — the ``state``
|
|
key simply reports ``"cancelled"``.
|
|
|
|
The cancel is COOPERATIVE, best-effort (phase 134 design): while a
|
|
run is in flight, ``POST /api/sync/cancel`` (the router's
|
|
``require_admin`` dependency applies — 403 anonymous) sets the
|
|
module-level ``_cancel_event`` AND ``_task.cancel()`` (unblocking
|
|
whatever ``await`` the pipeline is parked on — e.g. a per-file LLM
|
|
call inside the import), answering 202 the moment the request is
|
|
REGISTERED (the run may already be terminal by the next poll — the
|
|
status object, not the task, is authoritative). ``start_sync`` clears
|
|
the event when it starts a run, so a stale event from a previously
|
|
cancelled run can never bleed into the next one. ``_run_sync``
|
|
checks the event at the pipeline boundaries via
|
|
:func:`_check_cancelled` (raises ``asyncio.CancelledError`` when
|
|
set): the top of the per-source loop (before each clone/pull or
|
|
local-dir check), immediately before ``import_sources``, and
|
|
immediately before the overview + summaries span (the
|
|
``added + updated > 0`` block). Phase 194 (task 02) adds one more
|
|
await INSIDE the import span for the same shape: the pre-scan runs in
|
|
a ``to_thread`` worker, so a cancel parked there unblocks the RUN at
|
|
once (the ``cancelled`` terminal, nothing indexed) while the scan
|
|
thread finishes the session of its own — the phase-134/179 split, the
|
|
worker not being interruptible, pinned by
|
|
``tests/integration/test_sync_pending_count.py`` (importer side) and
|
|
this module's cancel-during-scan pin. Consequence a future reader must
|
|
know: a git subprocess already running at the cancel moment runs to
|
|
completion (Python threads are not interruptible — since phase 179
|
|
it occupies a ``to_thread`` worker rather than the loop, so it no
|
|
longer parks the EVENT LOOP while it does) — the cancel lands at the
|
|
NEXT boundary (after that source's git ops, between files, or before
|
|
overview/summaries); during the import span, the ``_task.cancel()``
|
|
from the endpoint interrupts at the next per-file await. Phase 179
|
|
changes only WHO waits: the run's await on the git worker is
|
|
unblocked immediately by ``_task.cancel()`` (so the run reaches that
|
|
next boundary) and every other request — the 2 s status poll
|
|
included — is served while the git thread finishes on its own. The
|
|
observable contract is phase 134's, unchanged: stop at the next
|
|
boundary, ``cancelled`` terminal, no corruption.
|
|
|
|
Phase 179 (task 02) — the sync's git operations run OFF the event
|
|
loop. Before it, ``clone_or_pull`` and ``file_commit_dates`` were
|
|
called as plain synchronous functions from inside ``_run_sync`` — the
|
|
coroutine ``POST /api/sync`` spawns with ``asyncio.create_task`` — so
|
|
their blocking ``subprocess.run`` (``scripts/git_sync.py``, ``run_git``)
|
|
parked the app's SINGLE uvicorn loop thread for the whole git op:
|
|
nothing else got scheduled, the ENTIRE HTTP surface stalled (the UI's
|
|
2 s ``GET /api/sync/status`` poll included) and the server resumed
|
|
only when git returned. That is the owner's ``TODO.md`` L3 suspicion
|
|
("a long git clone seems to freeze the server?"), CONFIRMED by the
|
|
in-process repro in ``tests/integration/test_sync_git_freeze.py`` — a
|
|
3 s stubbed clone made the status probe take ≈ 3.0 s (and
|
|
``GET /api/health`` too). Both git steps now go through
|
|
``asyncio.to_thread``, so the loop stays free for the whole clone
|
|
while the clone/pull and the commit-date walk occupy a worker thread.
|
|
Nothing else changes: the pipeline sequence, the error surfaces and
|
|
the cancel terminals above are exactly as documented — only the
|
|
SCHEDULING of the git calls moved (``scripts/git_sync.py`` itself is
|
|
untouched, and it stays the module that owns git spawning — since
|
|
phase 186 with exactly TWO sites there, the buffered ``run_git`` and
|
|
the streaming ``stream_git_lines``, both reached only through
|
|
``to_thread``). Known
|
|
BOUNDED non-fix (phase 179 decision A3): the run's short-lived
|
|
``SessionLocal()`` / ``effective_sources`` / version-bump calls stay
|
|
ON the loop — millisecond-scale SQLAlchemy, not the reported symptom,
|
|
deliberately out of scope.
|
|
|
|
The ``CancelledError`` contract has TWO paths (the former
|
|
"deliberately not caught" rule refined, not broken): in
|
|
``_run_sync``'s ``except asyncio.CancelledError`` clause (placed
|
|
before the ``except Exception``), a USER-requested cancel
|
|
(``_cancel_event`` set) lands the run in the ``cancelled`` terminal
|
|
+ one INFO line (PLAN §9: ``sync: cancelled by user state=cancelled
|
|
files_done=… files_total=…``) and then RE-RAISES (the task ends
|
|
cancelled — ``_task.done()`` is True, the status object stays
|
|
authoritative); an UNREQUESTED cancel (the app-shutdown path — the
|
|
event unset) re-raises UNCHANGED (the shutdown contract holds: a
|
|
background task must die in state, never as an unobserved exception
|
|
— swallowing an unrequested stop would mask a real one).
|
|
|
|
Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the
|
|
pipeline stage the run is in: the model-check + clone/pull prelude
|
|
reports ``null`` (the bare "Syncing…" label stays), then ``"scan"``
|
|
(phase 194, task 02 — set immediately before ``import_sources``,
|
|
REPLACING the pre-phase ``"import"`` set: what happens first in that
|
|
call is the pre-scan that predicts the change set, and the label the
|
|
task-03 branch renders for this phase is ``Scanning… (n/total)``), then
|
|
``"import"`` (flipped by
|
|
the FIRST per-file ``progress`` call while the phase is still
|
|
``"scan"`` — the flip lives in the runner's own ``_hook`` closure, so
|
|
the importer gains no extra callback; with
|
|
``BOR_SYNC_PENDING_COUNT=0`` no scan report fires at all and the flip
|
|
is simply the first file), ``"overview"`` (set
|
|
before ``regenerate_overview`` — a changed KB only), and
|
|
``"summaries"`` (set before a folder-summary generation — the
|
|
changed-KB incremental regeneration (phase 127, task 03 — the changed
|
|
set's folder keys only) or the unchanged-walk gap-fill; the no-gap
|
|
skip sets NO phase and the run stays ``"import"``). While in
|
|
``"summaries"`` the generator's ``on_progress`` hook fills
|
|
``current_summary`` (the ``source`` / ``source/folder_path`` being
|
|
summarized — the bare source name for the source-root row) plus
|
|
``summaries_done`` / ``summaries_total`` — the per-folder position
|
|
through the long summary span where the file count sits still (the
|
|
hook's ``total`` is the run's key count — the AFFECTED count on a
|
|
changed KB, the missing count on a gap-fill).
|
|
Terminal states (success, failed, and — phase 134 — cancelled) clear
|
|
``phase`` + ``current_summary`` but keep the run's final
|
|
``summaries_done`` / ``summaries_total`` (the phase-64
|
|
keep-final-counts convention).
|
|
|
|
Phase 129 (task 03) adds the in-flight file's EMBED-chunk position —
|
|
``current_chunk`` (the 1-based first chunk of the in-flight
|
|
token-budgeted batch) + ``chunks_total`` (the file's total content
|
|
chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed`
|
|
``on_progress`` hook (tasks 01/02). They are 0/0 before the first
|
|
batch (the prelude), during the ``"overview"`` / ``"summaries"``
|
|
phases, and in terminal states — the import phase clearing them at
|
|
EVERY point it ends (the ``phase = "overview"`` and
|
|
``phase = "summaries"`` transitions and every terminal, the same spots
|
|
``current_file`` clears). Unlike the phase-64 FILE counts, the chunk
|
|
position is a SUB-position of a file that's done — the keep-final-
|
|
counts convention does NOT extend to it: the file counts survive the
|
|
terminal for the result line's context, the chunk position clears
|
|
with the file.
|
|
|
|
Phase 152 (task 01) adds the in-flight file's VISION-image position —
|
|
``images_done`` (the 1-based index of the image about to be described
|
|
and embedded) + ``images_total`` (the file's total images for this
|
|
run — the three PDF vision spans of ONE file share ONE continuous
|
|
counter per file; task 02 wires the importer's
|
|
``on_image_progress`` hook, task 03 the runner's closure that feeds
|
|
these fields). They are 0/0 before the first image, during the
|
|
``"overview"`` / ``"summaries"`` phases, and in terminal states — the
|
|
same spots ``current_chunk`` clears. Unlike the chunk position (every
|
|
file embeds chunks), a file may have NO images at all, so the per-file
|
|
``progress`` hook ALSO resets them to 0/0: the pairing rule, extended
|
|
— a ``progress`` call names the file and clears the image position,
|
|
and every image position until the next ``progress`` call belongs to
|
|
it (a file with no images must not inherit the previous PDF's
|
|
position). ASSUMPTION A5: no new ``phase`` value — the image spans
|
|
live inside ``phase: "import"``.
|
|
|
|
Phase 187 (task 02) makes the run's MEMORY a logged number: every
|
|
lifecycle line gains trailing RSS fields read through
|
|
:mod:`app.core.memory` — ``sync: started … rss_mb=N`` and, in every
|
|
terminal, ``… rss_mb=N rss_peak_mb=M`` (the ``import: summary …`` line
|
|
carries ``rss_mb=`` too, on the importer side). APPEND-ONLY (LOCKED A2):
|
|
no existing field is renamed, reordered or re-typed, so every pre-phase
|
|
``grep`` keeps working; an unknowable reading renders ``rss_mb=?``
|
|
through the ONE :func:`app.core.memory.fmt_mb` helper. The numbers are
|
|
PROCESS numbers, not the host's cgroup figure — the
|
|
:mod:`app.core.memory` module docstring owns that caveat. The ``failed``
|
|
funnel keeps its ``logger.exception`` (the traceback) and gains ONE
|
|
sanitized one-line summary beside the status field, so the raw exception
|
|
string never reaches the log un-sanitized.
|
|
|
|
Phase 187 (task 03) puts the same numbers on the STATUS surface: the
|
|
status object carries ``rss_mb`` / ``rss_peak_mb`` (``int | null``,
|
|
sampled through the ONE :func:`_sample_rss` — at the run start, at every
|
|
``phase`` transition, in every terminal, and at most once per
|
|
``_RSS_SAMPLE_INTERVAL_S`` from the per-file progress hook, so a 27k-file
|
|
walk costs ~600 ``/proc`` reads and not 27k — LOCKED A4), and
|
|
``GET /api/sync/status`` reports both keys ADDITIVELY. Phase 194
|
|
(task 02) extends both halves of that rule to the new stage: the
|
|
``"scan"`` set before ``import_sources`` and the ``scan``→``import``
|
|
flip are phase TRANSITIONS and FORCE the read, while the pre-scan's own
|
|
per-file hook samples UNFORCED (throttled) — a 10,000-file scan costs
|
|
one ``/proc`` read per interval, never one per file. The terminals keep
|
|
the FINAL reading next to the final counts (the phase-64 convention), so
|
|
the page that started the run answers the incident's question without a
|
|
shell. A reading the platform cannot know stays ``null`` on the wire (the
|
|
UI renders no clause for it) — never 0, never an invented number.
|
|
|
|
The ``failed`` state's ``error`` string is masked by the shared
|
|
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
|
|
(imported here under the private name ``_sanitize_error``).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import logging
|
|
import time
|
|
from dataclasses import dataclass, field
|
|
from datetime import UTC, datetime
|
|
from pathlib import Path
|
|
from typing import Any, Literal
|
|
|
|
from fastapi import APIRouter, Depends, HTTPException
|
|
|
|
from app.config import get_settings
|
|
from app.core.auth import require_admin
|
|
from app.core.errors import sanitize_error as _sanitize_error
|
|
from app.core.memory import fmt_mb, rss_mb, rss_peak_mb
|
|
from app.db import SessionLocal
|
|
from app.rag.folder_summaries import generate_folder_summaries, missing_folder_summaries
|
|
from app.rag.git_sources import clone_url_for, effective_sources
|
|
from app.rag.importer import ImportSummary, import_sources
|
|
from app.rag.llm import LLMClient, check_models
|
|
from app.rag.overview import regenerate_overview
|
|
from app.rag.sources_meta import bump_sources_version, current_sources_version
|
|
from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates
|
|
from scripts.import_docs import repo_name
|
|
|
|
logger = logging.getLogger("app.api.sync")
|
|
|
|
router = APIRouter(
|
|
prefix="/sync",
|
|
tags=["sync"],
|
|
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
|
|
)
|
|
|
|
|
|
@dataclass
|
|
class SyncStatus:
|
|
"""In-memory state of the (at most one) in-flight sync run.
|
|
|
|
``state`` is a five-state machine: ``idle`` (never run / reset),
|
|
``running``, ``success``, ``failed``, and ``cancelled`` (phase 134,
|
|
task 01 — terminal like ``success`` / ``failed``: ``finished_at``
|
|
set, ``current_file`` / ``phase`` / ``current_summary`` cleared, the
|
|
final file + summary counts kept, but ``error = None`` and
|
|
``detail = {}`` — a cancel is intent, not failure). Terminal states
|
|
carry the run's ``detail`` (success) or ``error`` (failure) so the UI
|
|
can render the last result after a page reload (task 02's re-attach
|
|
behavior).
|
|
|
|
Phase 64 (task 02) progress fields: ``current_file`` is the
|
|
``source/relative/path`` the import is processing right now (null
|
|
outside the import phase — clone/pull first, terminal states
|
|
after); ``files_done`` / ``files_total`` carry the hook's
|
|
done/total position and survive a terminal state (the run's last
|
|
position is useful context next to the error). Since phase 194 the
|
|
pair is the hook's PENDING position (phase 194: the pre-scan's
|
|
predicted change set — ``files_total`` the files that will actually
|
|
be re-indexed), not the walk position; ``BOR_SYNC_PENDING_COUNT=0``
|
|
restores the walk position and every other caller (the CLI, the
|
|
upload job) is unaffected.
|
|
|
|
Phase 194 (task 02) scan fields: ``scan_done`` / ``scan_total``
|
|
carry the pre-scan's position — the 1-based file whose hash+read
|
|
just completed over the combined importable count the scan walks
|
|
(the importer's ``on_scan_progress`` hook). They are 0/0 before the
|
|
scan starts and on the ``BOR_SYNC_PENDING_COUNT=0`` rollback (no
|
|
scan, so no report), reset with the run like every other field, and
|
|
KEPT at every terminal beside the final file counts (the phase-64
|
|
keep-final-values convention — the scan the owner watched is still
|
|
on the page after the run). ``current_file`` stays ``null`` through
|
|
the phase: the scan names no file, it only counts. The degenerate
|
|
empty knowledge base is the honest edge — the scan predicts nothing
|
|
so no report fires and no ``progress`` call ever arrives: the phase
|
|
stays ``"scan"`` until the next transition or the terminal (which
|
|
clears it), and the label is the bare phase name with no count part
|
|
(the UI's ``total === 0`` guard).
|
|
|
|
Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage
|
|
(null in the model-check + clone/pull prelude and in terminal
|
|
states — the module docstring's phase machine, whose stages are now
|
|
``scan`` → ``import`` → ``overview`` → ``summaries``); while in the
|
|
``"summaries"`` phase, ``current_summary`` is the source /
|
|
``source/folder_path`` the generator is summarizing right now and
|
|
``summaries_done`` / ``summaries_total`` carry the generator's
|
|
progress hook's position — the hook's ``total`` is the run's key
|
|
count: the AFFECTED count on a changed KB (phase 127, task 03 —
|
|
the changed set's folder keys, NOT the table's candidate count) or
|
|
the missing count on an unchanged-walk gap-fill (as before).
|
|
Terminal states clear ``phase`` + ``current_summary`` but keep the
|
|
run's final summary counts (the phase-64 keep-final-counts
|
|
convention).
|
|
|
|
Phase 129 (task 03) chunk-position fields: ``current_chunk`` is
|
|
the 1-based first chunk of the in-flight token-budgeted batch and
|
|
``chunks_total`` the in-flight file's total content chunks (batch
|
|
granularity, the ``LLMClient.embed`` hook — the module docstring's
|
|
contract). They are 0/0 before the first batch, during the
|
|
``"overview"`` / ``"summaries"`` phases, and in terminal states —
|
|
the import phase clears them at every point it ends (the
|
|
overview/summaries transitions + every terminal, where
|
|
``current_file`` also clears). The phase-64 keep-final-counts
|
|
convention covers the FILE counts only — the chunk position is a
|
|
sub-position of a file that's done, so it clears in the terminal
|
|
while ``files_done`` / ``files_total`` keep their final values.
|
|
|
|
Phase 152 (task 01) image-position fields: ``images_done`` is
|
|
the 1-based index of the vision image about to be described and
|
|
``images_total`` the in-flight file's total images for this run
|
|
(the importer's ``on_image_progress`` hook — the three PDF vision
|
|
spans of one file share ONE continuous counter, the module
|
|
docstring's contract). Like the chunk position, the image
|
|
position is a SUB-position of a file that's done: the
|
|
keep-final-counts convention covers the FILE counts only, so it
|
|
clears with the file — 0/0 before the first image, during the
|
|
``"overview"`` / ``"summaries"`` phases, and in terminal states
|
|
(the same spots the chunk position clears) — and the per-file
|
|
``progress`` hook resets it to 0/0 (a ``progress`` call names the
|
|
file and clears the image position; a file with no images must
|
|
not inherit the previous PDF's position).
|
|
|
|
Phase 187 (task 03) memory fields: ``rss_mb`` is the process's
|
|
CURRENT resident set size in MB and ``rss_peak_mb`` its HIGH-WATER
|
|
mark — both PROCESS readings from :mod:`app.core.memory`, never the
|
|
host's cgroup figure (that one counts page cache; the caveat lives
|
|
in that module). They are written by ONE sampler, :func:`_sample_rss`:
|
|
forced at the run start, at every ``phase`` transition and in every
|
|
terminal, and throttled to one read per ``_RSS_SAMPLE_INTERVAL_S``
|
|
from the per-file ``progress`` hook (LOCKED A4 — never one ``/proc``
|
|
read per file). Both reset to ``None`` with the run — an idle or
|
|
never-run status reports null, and the Sources page renders no peak
|
|
clause for null — and every terminal KEEPS the final reading next to
|
|
the final counts (the phase-64 keep-final-values convention, the one
|
|
that makes "how much RAM did that sync use?" answerable from the
|
|
page after the run, not only from the log).
|
|
"""
|
|
|
|
state: Literal["idle", "running", "success", "failed", "cancelled"] = "idle"
|
|
started_at: datetime | None = None
|
|
finished_at: datetime | None = None
|
|
detail: dict[str, Any] = field(default_factory=dict)
|
|
error: str | None = None
|
|
# Phase 64 (task 02): per-file progress — the file the import is
|
|
# processing right now and the hook's done/total position.
|
|
current_file: str | None = None
|
|
files_done: int = 0
|
|
files_total: int = 0
|
|
# Phase 194 (task 02): the PRE-SCAN's position — the 1-based file
|
|
# whose hash+read just completed and the combined importable count
|
|
# the scan walks (the importer's on_scan_progress hook). 0/0 before
|
|
# the scan starts (and on the BOR_SYNC_PENDING_COUNT=0 rollback);
|
|
# reset with the run and kept at every terminal beside the final
|
|
# file counts (the phase-64 keep-final-values convention).
|
|
scan_done: int = 0
|
|
scan_total: int = 0
|
|
# Phase 98 (task 01): the phase machine — the pipeline stage the
|
|
# run is in and, while in the folder-summary phase, the folder
|
|
# being summarized plus the hook's done/total position. Phase 194
|
|
# (task 02) adds the "scan" stage (the pre-scan that predicts the
|
|
# run's change set — the module docstring's phase machine).
|
|
phase: Literal["scan", "import", "overview", "summaries"] | None = None
|
|
current_summary: str | None = None
|
|
summaries_done: int = 0
|
|
summaries_total: int = 0
|
|
# Phase 129 (task 03): the current file's EMBED chunk position —
|
|
# the first chunk of the in-flight token-budgeted batch and the
|
|
# file's total content chunks (the LLMClient.embed hook, batch
|
|
# granularity). 0/0 before the first batch, during the
|
|
# prelude/overview/summaries phases, and in terminal states.
|
|
current_chunk: int = 0
|
|
chunks_total: int = 0
|
|
# Phase 152 (task 01): the in-flight file's VISION-image position —
|
|
# the 1-based index of the image about to be described and the
|
|
# file's total images for this run (the importer's
|
|
# on_image_progress hook; the three PDF vision spans share ONE
|
|
# continuous counter per file). 0/0 before the first image, during
|
|
# the prelude/overview/summaries phases, and in terminal states.
|
|
images_done: int = 0
|
|
images_total: int = 0
|
|
# Phase 187 (task 03): the run's PROCESS memory in MB — current and
|
|
# high-water (app.core.memory, never the host's cgroup figure). None
|
|
# idle / never-run and reset with the run; kept as the FINAL reading
|
|
# in every terminal. Written ONLY through _sample_rss, the one
|
|
# sampling point (LOCKED A4).
|
|
rss_mb: int | None = None
|
|
rss_peak_mb: int | None = None
|
|
|
|
|
|
_status = SyncStatus()
|
|
_task: asyncio.Task[None] | None = None
|
|
# Phase 134 (task 01): the cancel request — set by POST
|
|
# /api/sync/cancel while a run is in flight; _run_sync's boundaries
|
|
# + its ``except CancelledError`` clause read it (the module
|
|
# docstring's cancel paragraph). Cleared by start_sync when a run
|
|
# starts, so a stale event can never bleed into the next run.
|
|
_cancel_event = asyncio.Event()
|
|
|
|
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hooks may
|
|
#: re-read the process memory, in seconds. Those hooks fire once per
|
|
#: file — the import's ``progress`` AND, since phase 194 (task 02), the
|
|
#: pre-scan's ``on_scan_progress`` — so without this guard a 27k-file
|
|
#: walk would cost 27k ``/proc/self/status`` reads; with it the same walk
|
|
#: costs ~600 and the operator watching the page still sees the number
|
|
#: move (PLAN §7.4's never-stale contract). Every other sampling point
|
|
#: (the run start, the ``phase`` transitions including the ``scan`` set
|
|
#: and its flip to ``import``, the terminals) FORCES the read instead.
|
|
_RSS_SAMPLE_INTERVAL_S = 5.0
|
|
|
|
#: ``time.monotonic()`` of the last RSS read; ``-inf`` means "never", so
|
|
#: the first throttled call reads instead of waiting out the interval.
|
|
#: Module-global like ``_status`` itself (the tests' fresh-state fixture
|
|
#: re-seeds both).
|
|
_rss_sampled_at: float = float("-inf")
|
|
|
|
|
|
def _sample_rss(force: bool = False) -> None:
|
|
"""Read the process memory onto ``_status`` — the ONE sampling point
|
|
(phase 187, task 03).
|
|
|
|
``force=True`` reads NOW: the run start, each ``phase`` transition,
|
|
every terminal — the points where the number is worth a syscall and
|
|
where a stale reading would be a lie. ``force=False`` (the per-file
|
|
progress hook) is a NO-OP inside :data:`_RSS_SAMPLE_INTERVAL_S` of
|
|
the previous read, which is what makes the sampling throttled rather
|
|
than per-file (LOCKED A4).
|
|
|
|
Both fields come off the SAME read, so the pair the status endpoint
|
|
and the lifecycle line report is one instant, not two. Nothing here
|
|
raises: :mod:`app.core.memory` degrades a platform that cannot know
|
|
to ``None``, which the status renders as JSON null (no UI clause) and
|
|
the log as ``?``.
|
|
|
|
**The pair is normalized to one truth: ``peak >= current``.** The two
|
|
numbers come from two different kernel counters read at two instants
|
|
(``VmRSS`` from ``/proc/self/status``, then ``ru_maxrss``), and the
|
|
kernel refreshes its high-water LAZILY, so a growing process really
|
|
can report a current reading ABOVE the reported high-water (measured
|
|
on this repo's own suite: ``rss_mb=164`` against ``rss_peak_mb=162``).
|
|
A "peak" below the run's own current footprint is not a smaller
|
|
number, it is a FALSE one — the high-water is by definition at least
|
|
the current set — so the ONE sampling point raises it to the current
|
|
reading. That is a lower bound on the truth, never an invention, and
|
|
it is why every consumer (both lifecycle lines, the status payload,
|
|
the page's clause) can now take ``peak >= current`` as an invariant
|
|
instead of carrying a slack. A ``None`` is still a ``None``: an
|
|
unknowable peak is never fabricated from the current reading.
|
|
"""
|
|
global _rss_sampled_at
|
|
now = time.monotonic()
|
|
if not force and (now - _rss_sampled_at) < _RSS_SAMPLE_INTERVAL_S:
|
|
return
|
|
_rss_sampled_at = now
|
|
current = rss_mb()
|
|
peak = rss_peak_mb()
|
|
if current is not None and peak is not None and peak < current:
|
|
peak = current # a high-water below the current set is a stale one
|
|
_status.rss_mb = current
|
|
_status.rss_peak_mb = peak
|
|
|
|
|
|
@router.get("/status")
|
|
def sync_status() -> dict[str, Any]:
|
|
"""Current sync state (the UI polls this every 2 s — task 02).
|
|
|
|
``started_at`` / ``finished_at`` are ISO-8601 strings or null.
|
|
``current_file`` (phase 64) is the ``source/relative/path`` the
|
|
import is processing right now — null during the clone/pull phase
|
|
and in terminal states; ``files_done`` / ``files_total`` carry the
|
|
hook's position (0/0 idle) and keep their final values in the
|
|
terminal — the position in the pre-scan's PREDICTED change set on a
|
|
progress caller (phase 194), the walk position under
|
|
``BOR_SYNC_PENDING_COUNT=0``. ``scan_done`` / ``scan_total`` (phase
|
|
194) are the PRE-SCAN's position: the 1-based file whose
|
|
hash+compare just ran over the combined importable count the scan
|
|
walks. They are 0/0 before the scan starts — the prelude, a run on
|
|
the ``0`` rollback, and every run that never happened — and in a
|
|
terminal they keep the FINAL scan values beside the final file
|
|
counts (the phase-64 keep-final-values convention); ``current_file``
|
|
stays null while ``phase`` is ``"scan"``, because the scan names no
|
|
file. Both keys are ADDITIVE — placed after ``files_total``, every
|
|
pre-existing key keeping its name, type and position. The degenerate
|
|
empty knowledge base reports the phase without ever reporting a
|
|
position (0/0 — the UI's ``total === 0`` guard renders no count
|
|
part). ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
|
in-flight file's embed-chunk position (batch granularity — the
|
|
first chunk of the in-flight batch over the file's total content
|
|
chunks): 0/0 before the first batch, in the prelude/overview/
|
|
summaries phases, and in terminal states (they clear at every
|
|
point the import phase ends; the FILE counts do not). ``phase``
|
|
(phase 98) is the pipeline stage — null in the prelude and
|
|
terminal states; while in ``"summaries"``, ``current_summary`` +
|
|
``summaries_done`` / ``summaries_total`` carry the generator's
|
|
position (the terminal keeps the final summary counts).
|
|
|
|
Phase 134 (task 01): ``state`` also reports ``"cancelled"`` for a
|
|
user-cancelled run — a terminal state with ``error`` null and
|
|
``detail`` empty; the run's final file + summary counts are kept,
|
|
the chunk position cleared.
|
|
|
|
Phase 152 (task 01): ``images_done`` / ``images_total`` are the
|
|
in-flight file's vision-image position (the importer's
|
|
``on_image_progress`` hook — the three PDF vision spans of one
|
|
file share ONE continuous counter): 0/0 before the first image, in
|
|
the prelude/overview/summaries phases, and in terminal states; the
|
|
per-file ``progress`` hook clears it at every file boundary (a
|
|
file with no images never shows a stale position).
|
|
|
|
Phase 187 (task 03): ``rss_mb`` / ``rss_peak_mb`` are the run's
|
|
PROCESS memory in MB — the current resident set and the high-water
|
|
mark (:mod:`app.core.memory`, never the host's cgroup figure, which
|
|
counts page cache). ``null`` means "no reading", which is the idle /
|
|
never-run state AND any platform that cannot know — never 0, and
|
|
never an invented number. The run samples them through
|
|
:func:`_sample_rss`: at the run start, at every ``phase``
|
|
transition, throttled to one read per 5 s through the import's
|
|
per-file hook, and FORCED in every terminal — so a terminal reports
|
|
the run's FINAL reading next to its final counts (the phase-64
|
|
convention, same as ``files_done`` / ``files_total``), and a reload
|
|
after the run still shows the number that ``sync: done …`` logged.
|
|
Additive only — every pre-existing key keeps its name, type, and
|
|
meaning.
|
|
"""
|
|
return {
|
|
"state": _status.state,
|
|
"started_at": _status.started_at.isoformat() if _status.started_at else None,
|
|
"finished_at": _status.finished_at.isoformat() if _status.finished_at else None,
|
|
"detail": _status.detail,
|
|
"error": _status.error,
|
|
"current_file": _status.current_file,
|
|
"files_done": _status.files_done,
|
|
"files_total": _status.files_total,
|
|
# Phase 194 (task 02): the pre-scan's position (the importer's
|
|
# on_scan_progress hook) — 0/0 before the scan starts and on the
|
|
# BOR_SYNC_PENDING_COUNT=0 rollback, the FINAL pair kept in every
|
|
# terminal beside the file counts above (the keep-final-values
|
|
# convention). Additive: every pre-existing key keeps its name,
|
|
# type and position.
|
|
"scan_done": _status.scan_done,
|
|
"scan_total": _status.scan_total,
|
|
# Phase 129 (task 03): the in-flight file's embed-chunk
|
|
# position — the dataclass defaults make idle 0/0, and the
|
|
# runner clears it at every point the import phase ends
|
|
# (overview/summaries transitions + every terminal).
|
|
"current_chunk": _status.current_chunk,
|
|
"chunks_total": _status.chunks_total,
|
|
# Phase 152 (task 01): the in-flight file's VISION-image
|
|
# position (the importer's on_image_progress hook — the three
|
|
# PDF vision spans share ONE continuous counter per file):
|
|
# 0/0 before the first image, during the
|
|
# prelude/overview/summaries phases, and in terminal states
|
|
# (cleared at every point the import phase ends; the per-file
|
|
# progress hook re-clears it at every file boundary).
|
|
"images_done": _status.images_done,
|
|
"images_total": _status.images_total,
|
|
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
|
|
# dataclass defaults) and in terminal states (which clear
|
|
# phase + current_summary but keep the final summary counts).
|
|
"phase": _status.phase,
|
|
"current_summary": _status.current_summary,
|
|
"summaries_done": _status.summaries_done,
|
|
"summaries_total": _status.summaries_total,
|
|
# Phase 187 (task 03): the run's PROCESS memory in MB (current +
|
|
# high-water), read through the ONE _sample_rss. null idle /
|
|
# never-run; in a terminal it is the run's FINAL reading (the
|
|
# phase-64 keep-final-values convention — the number the
|
|
# 2026-10-15 incident had to reconstruct from systemd teardown
|
|
# lines, now on the page that started the run).
|
|
"rss_mb": _status.rss_mb,
|
|
"rss_peak_mb": _status.rss_peak_mb,
|
|
}
|
|
|
|
|
|
@router.post("", status_code=202)
|
|
async def start_sync() -> dict[str, str]:
|
|
"""Start the clone → import → overview sync as a background task.
|
|
|
|
202 + ``sync started`` kicks off :func:`_run_sync` on the app's event
|
|
loop. 409 when a run is already in flight (one sync at a time — the
|
|
status endpoint is the single source of truth for the run, and the
|
|
UI re-attaches to it rather than starting a second one).
|
|
|
|
Phase 134 (task 01): the run's cancel request resets with the run —
|
|
``_cancel_event.clear()`` before the task is created, so a stale
|
|
event from a previously cancelled run can never bleed into this one.
|
|
"""
|
|
global _task
|
|
if _task is not None and not _task.done():
|
|
raise HTTPException(status_code=409, detail="a sync is already running")
|
|
_cancel_event.clear()
|
|
_task = asyncio.create_task(_run_sync())
|
|
return {"detail": "sync started"}
|
|
|
|
|
|
@router.post("/cancel", status_code=202)
|
|
async def cancel_sync() -> dict[str, str]:
|
|
"""Request the cancel of the in-flight run (phase 134, task 01).
|
|
|
|
202 + ``cancel requested`` is answered the moment the request is
|
|
REGISTERED: the module ``_cancel_event`` is set (the runner's
|
|
``_check_cancelled`` boundaries + its ``except CancelledError``
|
|
conversion see it) AND ``_task.cancel()`` unblocks whatever
|
|
``await`` the pipeline is parked on (e.g. a per-file LLM call
|
|
inside the import). The run may already be terminal by the next
|
|
status poll — the status object, not the task, is authoritative,
|
|
so the UI's poll settles on the terminal (``cancelled``) state.
|
|
409 ``no sync is running`` when there is no run in flight (the
|
|
``_task`` is None or done — the harmless race the UI's poll
|
|
settles; no error copy, no state change).
|
|
"""
|
|
if _task is None or _task.done():
|
|
raise HTTPException(status_code=409, detail="no sync is running")
|
|
_cancel_event.set()
|
|
_task.cancel()
|
|
return {"detail": "cancel requested"}
|
|
|
|
|
|
def _check_cancelled() -> None:
|
|
"""Phase 134 (task 01): the cooperative cancel boundary.
|
|
|
|
Raises ``asyncio.CancelledError`` when the module ``_cancel_event``
|
|
is set (the user requested a stop — ``POST /api/sync/cancel``);
|
|
:func:`_run_sync`'s ``except asyncio.CancelledError`` clause
|
|
converts it into the ``cancelled`` terminal state. Called at the
|
|
pipeline boundaries (the module docstring's cancel paragraph) —
|
|
the top of the per-source loop, before ``import_sources``, and
|
|
before the overview + summaries span. A git subprocess in flight
|
|
at the cancel moment runs to completion in its ``to_thread``
|
|
worker (phase 179: threads are not interruptible, but the loop is
|
|
no longer parked by it); the cancel lands at the NEXT boundary —
|
|
or, when ``_task.cancel()`` interrupts the ``to_thread`` await
|
|
itself, right there, with the worker finishing on its own.
|
|
"""
|
|
if _cancel_event.is_set():
|
|
raise asyncio.CancelledError()
|
|
|
|
|
|
async def _run_sync() -> None:
|
|
"""The full sync pipeline, one in-process background task.
|
|
|
|
Every failure mode (git, embeddings, anything else) lands in the
|
|
``failed`` state with a sanitized ``error`` string — a background
|
|
task must die in state, never as an unobserved exception.
|
|
|
|
Cancellation (phase 134, task 01 — the module docstring's cancel
|
|
paragraph): the cooperative boundaries (:func:`_check_cancelled`
|
|
at the top of the per-source loop, before ``import_sources``, and
|
|
before the overview + summaries span) plus the ``except
|
|
asyncio.CancelledError`` clause convert a USER-requested cancel
|
|
(``_cancel_event`` set by ``POST /api/sync/cancel``) into the
|
|
``cancelled`` terminal state + one INFO line, then re-raise (the
|
|
task ends cancelled — the status object stays authoritative). An
|
|
UNREQUESTED cancel (the app-shutdown path — the event unset)
|
|
re-raises unchanged: a background task is never swallowed, and
|
|
swallowing a real stop would mask it.
|
|
|
|
Phase 179 (task 02): the per-source git steps (``clone_or_pull``,
|
|
``file_commit_dates``) run off the loop via ``asyncio.to_thread``
|
|
so a long clone no longer stalls the rest of the HTTP surface (the
|
|
module docstring's freeze paragraph); the run's own sequencing is
|
|
byte-for-byte the pipeline above.
|
|
|
|
Phase 194 (task 02): the phase set before ``import_sources`` is
|
|
``"scan"`` (not ``"import"``), because that call begins with the
|
|
off-loop pre-scan that predicts the change set; the FIRST per-file
|
|
``progress`` call flips it to ``"import"`` inside ``_hook``. Both
|
|
transitions force the memory sample, the scan's own per-file hook
|
|
does not (LOCKED A4), and ``scan_done`` / ``scan_total`` reset with
|
|
the run and survive every terminal beside the file counts.
|
|
"""
|
|
_status.state = "running"
|
|
_status.started_at = datetime.now(UTC)
|
|
_status.finished_at = None
|
|
_status.detail = {}
|
|
_status.error = None
|
|
# Phase 64 (task 02): the progress fields reset with the run — no
|
|
# current file until the import starts (the clone/pull phase).
|
|
_status.current_file = None
|
|
_status.files_done = 0
|
|
_status.files_total = 0
|
|
# Phase 194 (task 02): the pre-scan's position resets with the run —
|
|
# 0/0 until the scan reports its first file (and 0/0 for the whole
|
|
# run on the BOR_SYNC_PENDING_COUNT=0 rollback, which scans nothing).
|
|
_status.scan_done = 0
|
|
_status.scan_total = 0
|
|
# Phase 98 (task 01): the phase fields reset with the run — no
|
|
# phase until the import starts (the model-check + clone/pull
|
|
# prelude reports null), no folder until the summary span starts.
|
|
_status.phase = None
|
|
_status.current_summary = None
|
|
_status.summaries_done = 0
|
|
_status.summaries_total = 0
|
|
# Phase 129 (task 03): the chunk-position fields reset with the
|
|
# run — 0/0 until the import's first embed batch reports.
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image-position fields reset with the
|
|
# run — 0/0 until the import's first vision span reports (the
|
|
# per-file progress hook re-clears them at every file boundary).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
# Phase 187 (task 03): the run-start memory sample — forced, so it
|
|
# BOTH replaces the previous run's readings (a forced sample fully
|
|
# determines the pair; None where the platform cannot know) and
|
|
# re-arms the 5 s throttle. No stale peak ever bleeds into a new
|
|
# run's status.
|
|
_sample_rss(force=True)
|
|
try:
|
|
settings = get_settings()
|
|
# Step 1 (phase 41): fail fast — verify both models the sync
|
|
# needs (embed + summary) before source resolution or any
|
|
# clone. The client is reused for the import + overview below.
|
|
llm = LLMClient()
|
|
await check_models(llm)
|
|
# The background task has no request session: open a short-lived
|
|
# one around the shared phase-35/38 resolver (DB rows of both
|
|
# kinds win; the BOR_GIT_SOURCES git list is a fallback while
|
|
# the table is empty).
|
|
db = SessionLocal()
|
|
try:
|
|
rows, origin = effective_sources(db)
|
|
finally:
|
|
db.close()
|
|
if not rows:
|
|
# The button targets the admin-managed source registry
|
|
# (manual --source dirs have no repo to clone) — an empty
|
|
# config on *both* origins (no git rows, no local rows, no
|
|
# env URLs) fails loudly instead of silently importing the
|
|
# legacy directories.
|
|
raise GitSyncError("no sources configured (git or local)")
|
|
git_count = sum(1 for row in rows if row.kind == "git")
|
|
# Phase 187 (task 02): the trailing rss_mb= is APPEND-ONLY (LOCKED
|
|
# A2) — every pre-phase grep of this line keeps matching. Task 03
|
|
# logs the run-start SAMPLE (the reading the status reports) so
|
|
# the line and the payload never disagree — the shape is
|
|
# byte-identical to task 02's.
|
|
logger.info(
|
|
"sync: started repos=%d origin=%s git=%d local=%d rss_mb=%s",
|
|
len(rows), origin, git_count, len(rows) - git_count,
|
|
fmt_mb(_status.rss_mb),
|
|
)
|
|
sources_root = Path(settings.sources_dir).expanduser()
|
|
sources: list[Path] = []
|
|
ignore_by_root: dict[str, list[str]] = {}
|
|
include_hidden_by_root: dict[str, bool] = {}
|
|
doc_dates_by_root: dict[str, dict[str, datetime]] = {}
|
|
for row in rows:
|
|
# Phase 134 (task 01): the per-source boundary — a cancel
|
|
# registered while the previous source's git ops run lands
|
|
# here, before this source's clone/pull or local-dir check.
|
|
# Phase 179 (task 02): those git ops now run in a
|
|
# ``to_thread`` worker, so a cancel parked on them no
|
|
# longer means a PARKED EVENT LOOP — ``_task.cancel()``
|
|
# unblocks this run at the ``await`` and the loop serves
|
|
# everything else immediately, while the in-flight git
|
|
# thread runs its subprocess to completion (Python threads
|
|
# are not interruptible). Contract unchanged (phase 134):
|
|
# stop at the next boundary, ``cancelled`` terminal.
|
|
_check_cancelled()
|
|
if row.kind == "git":
|
|
# Phase 121: the token column is injected into the clone
|
|
# URL ONLY here (clone_url_for — NULL token → the bare
|
|
# stored URL verbatim); repo_name stays on the bare URL
|
|
# so the checkout directory name is credential-free.
|
|
# Phase 135: the row's branch pin is the third
|
|
# argument — NULL (every pre-phase-135 row + every
|
|
# env-fallback row) keeps the pre-phase argv
|
|
# byte-identical; a pinned branch is checked out on a
|
|
# fresh clone (--branch) and guarded + tracked on an
|
|
# existing checkout (fetch + ff-merge of that ref).
|
|
# Phase 179 (task 02): OFF the event loop — the
|
|
# blocking subprocess inside clone_or_pull used to
|
|
# stall the whole HTTP surface for the duration of the
|
|
# clone, the UI's 2 s GET /api/sync/status poll first
|
|
# among the victims; to_thread keeps the loop serving
|
|
# while the git worker runs.
|
|
root = await asyncio.to_thread(
|
|
clone_or_pull,
|
|
clone_url_for(row),
|
|
sources_root / repo_name(row.url),
|
|
row.branch,
|
|
)
|
|
# Phase 106 (D2): the checkout's per-file last-commit
|
|
# dates, keyed by the SAME root string the importer
|
|
# sees (full-history checkouts → true per-file
|
|
# last-commit dates for every git source — phase 107
|
|
# D11 supersedes phase 106 D10's shallow tip-date
|
|
# behavior). Local rows contribute nothing — the
|
|
# importer's mtime fallback applies to them.
|
|
# Phase 179 (task 02): off the loop for the same
|
|
# reason — a full-history ``git log`` walk over a big
|
|
# checkout is as blocking as the clone was.
|
|
doc_dates_by_root[str(root)] = await asyncio.to_thread(
|
|
file_commit_dates, root
|
|
)
|
|
else:
|
|
# kind=local — the stored expanded path (phase 38 also
|
|
# mirrors it in the NOT-NULL ``url`` location column, the
|
|
# ``or`` keeps the type checker honest); re-verified at
|
|
# sync time because the directory may have moved or been
|
|
# deleted since add-time.
|
|
root = Path(row.path or row.url).expanduser()
|
|
if not root.is_dir():
|
|
raise GitSyncError(f"local source missing: {root}")
|
|
sources.append(root)
|
|
# Phase 89: the row's ignore list, keyed by the SAME root
|
|
# string the importer sees; two rows sharing a root string
|
|
# get the union (extend, not replace) — the sibling/repo-name
|
|
# edge.
|
|
if row.ignore_paths:
|
|
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
|
|
# Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
|
|
# the SAME root string the importer sees; a shared-root
|
|
# collision ORs — if EITHER row says "index hidden", the
|
|
# root does (the ignore-map union's boolean mirror).
|
|
include_hidden_by_root[str(root)] = (
|
|
include_hidden_by_root.get(str(root), False)
|
|
or bool(row.include_hidden)
|
|
)
|
|
# Phase 64 (task 02): the per-file progress hook — the status
|
|
# endpoint reports the file being processed right now. The
|
|
# closure captures the module ``_status`` exactly like the state
|
|
# assignments above.
|
|
def _hook(source: str, rel: str, done: int, total: int) -> None:
|
|
_status.current_file = f"{source}/{rel}"
|
|
_status.files_done = done
|
|
_status.files_total = total
|
|
# Phase 152 (task 01): the pairing rule, extended — a
|
|
# progress call names the file AND clears the image
|
|
# position: every image position until the next progress
|
|
# call belongs to this file. A file with no images must
|
|
# not inherit the previous PDF's position (unlike the
|
|
# chunk position, not every file embeds chunks, so the
|
|
# run-start reset alone cannot cover it).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
# Phase 194 (task 02): the scan→import FLIP. The phase is
|
|
# set to "scan" immediately before import_sources, and the
|
|
# first file the walk actually reaches is where the running
|
|
# label stops being a scan — no extra callback reaches the
|
|
# importer, the flip lives in this existing per-file hook.
|
|
if _status.phase == "scan":
|
|
_status.phase = "import"
|
|
# A phase TRANSITION is a forced sampling point (phase
|
|
# 187), so this one read is not the throttled per-file
|
|
# one the tail of this closure does.
|
|
_sample_rss(force=True)
|
|
# Phase 187 (task 03): the run's memory, sampled THROTTLED
|
|
# (unforced) — this hook fires once per file, so a 27k-file
|
|
# walk reads /proc ~600 times instead of 27k (LOCKED A4)
|
|
# while the number on the page still climbs (PLAN §7.4).
|
|
_sample_rss()
|
|
|
|
# Phase 129 (task 03): the in-flight file's embed-chunk
|
|
# position — the LLMClient.embed hook (task 01) fires before
|
|
# each token-budgeted batch with (first_chunk_1based, total);
|
|
# the file's IDENTITY is the sibling ``progress`` hook's (the
|
|
# two hooks are paired — a progress call names the file, and
|
|
# every chunk position until the next one belongs to it).
|
|
def _chunk_hook(chunk: int, total: int) -> None:
|
|
_status.current_chunk = chunk
|
|
_status.chunks_total = total
|
|
|
|
# Phase 152 (task 03): the in-flight file's VISION-image
|
|
# position — the importer's ``on_image_progress`` hook (task
|
|
# 02) fires before each of the PDF's vision describe calls
|
|
# with (position_1based, total), the file's ONE continuous
|
|
# counter across its three vision spans (the importer owns
|
|
# the continuity — this closure only lands the pair); the
|
|
# file's IDENTITY is the sibling ``progress`` hook's, which
|
|
# ALSO resets this position (the three hooks are paired:
|
|
# ``progress`` names the file and clears the image position,
|
|
# ``on_chunk_progress`` reports the embed-batch position,
|
|
# ``on_image_progress`` reports the vision-image position).
|
|
def _image_hook(done: int, total: int) -> None:
|
|
_status.images_done = done
|
|
_status.images_total = total
|
|
|
|
# Phase 194 (task 02): the PRE-SCAN's position — the importer's
|
|
# ``on_scan_progress`` hook, fired once per importable file while
|
|
# the run's change set is still being PREDICTED (before the walk),
|
|
# with (position_1based, scan_total). It runs on the scan's
|
|
# WORKER thread, so this closure only assigns fields on the
|
|
# module's own status object — the house hook contract that makes
|
|
# each call a single GIL-atomic store (phase 179). The memory
|
|
# sample is UNFORCED here: this fires once per file, exactly like
|
|
# ``_hook`` above, so it costs one /proc read per interval and
|
|
# never one per file (LOCKED A4).
|
|
def _scan_hook(scan: int, total: int) -> None:
|
|
_status.scan_done = scan
|
|
_status.scan_total = total
|
|
_sample_rss()
|
|
|
|
# Phase 98 (task 01) + phase 194 (task 02): the phase machine —
|
|
# the SCAN phase starts NOW (the model-check + clone/pull prelude
|
|
# above reported ``phase: null``). What runs first inside
|
|
# ``import_sources`` is the pre-scan that predicts the change set,
|
|
# and the label the owner reads through it is "Scanning…"
|
|
# (phase 193's "import" set moved to the first ``_hook`` call
|
|
# above — the flip), so the count the page shows is never a scan
|
|
# wearing the work's shape.
|
|
_status.phase = "scan"
|
|
# Phase 187 (task 03): the phase transitions are the forced
|
|
# sampling points — each stage of the run begins with its own
|
|
# reading on the status (the per-file hook throttles).
|
|
_sample_rss(force=True)
|
|
# Phase 134 (task 01): the import boundary — a cancel between
|
|
# the source walk and the import (or parked in a blocking git
|
|
# span that just finished) lands here, before any file is
|
|
# imported.
|
|
_check_cancelled()
|
|
summary: ImportSummary = await import_sources(
|
|
sources,
|
|
llm,
|
|
prune=True,
|
|
progress=_hook,
|
|
on_chunk_progress=_chunk_hook,
|
|
on_image_progress=_image_hook, # phase 152 (task 03)
|
|
on_scan_progress=_scan_hook, # phase 194 (task 02)
|
|
ignore_by_root=ignore_by_root,
|
|
include_hidden_by_root=include_hidden_by_root,
|
|
doc_dates_by_root=doc_dates_by_root,
|
|
)
|
|
# Phase 134 (task 01): the overview + summaries boundary — a
|
|
# cancel during the import span (the endpoint's _task.cancel()
|
|
# interrupts at the next per-file await) or registered anywhere
|
|
# before this point lands here, before the lite spans start.
|
|
_check_cancelled()
|
|
overview = False
|
|
if summary.added + summary.updated > 0:
|
|
_status.phase = "overview" # phase 98 (task 01)
|
|
_sample_rss(force=True) # phase 187 (task 03)
|
|
# Phase 129 (task 03): the import phase ends here — the
|
|
# chunk position (a sub-position of a done file) clears
|
|
# with the phase change; the file counts keep their final
|
|
# values (the phase-64 convention covers files only).
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image position clears at the
|
|
# same point (a sub-position of a done file, like the
|
|
# chunk position).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
overview = await regenerate_overview(llm)
|
|
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
|
|
# the folder summaries — the drill-down ls's per-level
|
|
# descriptions. A changed KB (added + updated > 0) is an
|
|
# INCREMENTAL regeneration (phase 127, task 03): the run's
|
|
# changed documents (summary.changed_paths — the import's
|
|
# added/updated/pruned set) are passed as changed_docs= and
|
|
# ONLY their folder chains (each document's folder up to its
|
|
# source root) are rewritten — a one-file change burns a
|
|
# handful of rows, not the whole table (the CLI keeps the
|
|
# byte-identical full regeneration — it passes no change set);
|
|
# an unchanged re-sync takes the GAP probe instead of the old
|
|
# table-empty one — a candidate folder (at least 1 doc — every
|
|
# existing folder) with no stored row: a gap
|
|
# fills ONLY the missing rows (only_missing=True — the
|
|
# subsumed table-empty first-sync trigger included, where
|
|
# every candidate is missing), a complete table burns zero
|
|
# lite calls. The generator is per-folder fail-soft (a lite
|
|
# outage never flips the run to failed) and only flushes: this
|
|
# run's own short-lived session commits (the phase-53
|
|
# convention), so the step-6 bump stays change-gated on the KB,
|
|
# not on the summaries. The status surface is the PHASE
|
|
# machine (phase 98, task 01): the stats dict itself stays
|
|
# log-only (the detail shape is untouched).
|
|
# Phase 98 (task 01): the folder-summary progress hook — the
|
|
# closure-captures-``_status`` convention the file's ``_hook``
|
|
# above already uses; the generator's ``total`` (the loop-start
|
|
# key count — the affected count on a changed KB, the missing
|
|
# count on a gap-fill) and the bare source name for the root
|
|
# row come straight from the hook's arguments (D5's shape).
|
|
def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
|
|
_status.current_summary = (
|
|
source if folder_path == "" else f"{source}/{folder_path}"
|
|
)
|
|
_status.summaries_done = done
|
|
_status.summaries_total = total
|
|
|
|
fs_db = SessionLocal()
|
|
try:
|
|
if summary.added + summary.updated > 0:
|
|
# Changed KB: incremental regeneration (phase 127,
|
|
# task 03) — ONLY the changed set's folder keys
|
|
# (changed_docs=summary.changed_paths: each document's
|
|
# folder chain up to its source root); the summaries
|
|
# phase starts now, and the hook reports the per-folder
|
|
# position through the (long) span (its total is the
|
|
# affected count).
|
|
_status.phase = "summaries"
|
|
_sample_rss(force=True) # phase 187 (task 03)
|
|
# Phase 129 (task 03): the import phase ends here too
|
|
# (the changed-KB branch skips no phase) — same
|
|
# chunk-position clear as the overview transition.
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image position clears at
|
|
# the same point (a sub-position of a done file).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
folder_stats = await generate_folder_summaries(
|
|
fs_db, llm, changed_docs=summary.changed_paths,
|
|
on_progress=_summary_hook,
|
|
)
|
|
fs_db.commit()
|
|
logger.info(
|
|
"sync: folder_summaries changed=%d stats=%s",
|
|
len(summary.changed_paths), folder_stats,
|
|
)
|
|
else:
|
|
missing = missing_folder_summaries(fs_db)
|
|
if missing:
|
|
# Unchanged-walk gap-fill: the SAME summaries
|
|
# phase + hook (the fill's total is the missing
|
|
# count — the hook's loop-start ``total``).
|
|
_status.phase = "summaries"
|
|
_sample_rss(force=True) # phase 187 (task 03)
|
|
# Phase 129 (task 03): the import phase ends here
|
|
# too (the unchanged-walk gap-fill branch) — same
|
|
# chunk-position clear.
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image position clears
|
|
# at the same point (a sub-position of a done
|
|
# file).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
folder_stats = await generate_folder_summaries(
|
|
fs_db, llm, only_missing=True, on_progress=_summary_hook
|
|
)
|
|
fs_db.commit()
|
|
logger.info(
|
|
"sync: folder_summaries gap-fill missing=%d stats=%s",
|
|
len(missing), folder_stats,
|
|
)
|
|
else:
|
|
# The no-gap skip sets NO phase — the run stays
|
|
# "import" through to the terminal (the phase-64
|
|
# bare-label pin holds for this shape).
|
|
logger.info("sync: folder_summaries skipped (KB unchanged)")
|
|
finally:
|
|
fs_db.close()
|
|
# Phase 53 (task 02): a sync that changed the KB advances the
|
|
# sources version exactly once — the saved-chat invalidation
|
|
# marker (task 03 stamps rows against it). The gate is
|
|
# deliberately broader than the overview's above: a pruned
|
|
# document can invalidate a saved answer that cited it, so
|
|
# ``pruned > 0`` bumps too. The bump commits in its own short
|
|
# session (the ``effective_sources`` pattern above), so it
|
|
# lands even if the best-effort overview then fails — the index
|
|
# really did change. An unchanged re-sync never bumps; it
|
|
# reports the current generation instead, so the detail always
|
|
# carries the generation the KB is now at.
|
|
db = SessionLocal()
|
|
try:
|
|
if summary.added + summary.updated + summary.pruned > 0:
|
|
sources_version = bump_sources_version(db)
|
|
db.commit()
|
|
else:
|
|
sources_version = current_sources_version(db)
|
|
finally:
|
|
db.close()
|
|
_status.state = "success"
|
|
_status.finished_at = datetime.now(UTC)
|
|
_status.current_file = None # phase 64: keep the final counts
|
|
_status.phase = None # phase 98: keep the final summary counts
|
|
_status.current_summary = None
|
|
# Phase 194 (task 02): the scan position is NOT cleared here —
|
|
# like the file counts it keeps its final values beside them
|
|
# (the keep-final-values convention), so the page reloaded after
|
|
# the run still says how big the predicted change set was.
|
|
# Phase 129 (task 03): the chunk position is a sub-position of
|
|
# a file that's done — it clears in the terminal (unlike the
|
|
# FILE counts, which keep their final values for the result
|
|
# line's context).
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image position is a sub-position of
|
|
# a file that's done — it clears in the terminal too (the FILE
|
|
# counts keep their final values, the sub-position does not).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
_status.detail = {
|
|
"files": summary.files,
|
|
"added": summary.added,
|
|
"updated": summary.updated,
|
|
"unchanged": summary.unchanged,
|
|
"pruned": summary.pruned,
|
|
"errors": summary.errors,
|
|
"chunks": summary.chunks,
|
|
"summaries": summary.summaries,
|
|
"summary_errors": summary.summary_errors,
|
|
"dates_updated": summary.dates_updated,
|
|
"overview": overview,
|
|
"sources_version": sources_version,
|
|
}
|
|
# Phase 187 (task 02): the memory the run is at NOW and its
|
|
# high-water mark, next to the final counts (the phase-64
|
|
# keep-final-values convention, extended to the numbers the
|
|
# 2026-10-15 incident had to reconstruct from systemd teardown
|
|
# lines). Appended after ``detail=`` — the line stays greppable
|
|
# as ``sync: done detail=``. Task 03 forces the sample FIRST and
|
|
# logs it, so the payload's rss_mb / rss_peak_mb and this line
|
|
# carry the SAME final pair (the phase-64 convention, kept
|
|
# honest across both surfaces).
|
|
_sample_rss(force=True)
|
|
logger.info(
|
|
"sync: done detail=%s rss_mb=%s rss_peak_mb=%s",
|
|
_status.detail, fmt_mb(_status.rss_mb), fmt_mb(_status.rss_peak_mb),
|
|
)
|
|
except asyncio.CancelledError:
|
|
# Phase 134 (task 01): the TWO conversion paths (the module
|
|
# docstring's cancel paragraph). A user-requested cancel —
|
|
# ``_cancel_event`` set by POST /api/sync/cancel, whether the
|
|
# endpoint's ``_task.cancel()`` interrupted a parked await or a
|
|
# ``_check_cancelled`` boundary raised — lands the run in the
|
|
# ``cancelled`` terminal and re-raises (the task ends
|
|
# cancelled — ``_task.done()`` is True, the status object stays
|
|
# authoritative). An unrequested cancel (the app-shutdown path
|
|
# — the event unset) re-raises unchanged: never swallow a real
|
|
# stop.
|
|
if _cancel_event.is_set():
|
|
_status.state = "cancelled"
|
|
_status.finished_at = datetime.now(UTC)
|
|
_status.error = None # a cancel is intent, not failure
|
|
_status.current_file = None # keep the final counts
|
|
_status.phase = None # keep the final summary counts
|
|
_status.current_summary = None
|
|
# Phase 194 (task 02): the scan position survives the cancel
|
|
# too — where a cancel landed DURING the pre-scan it is the
|
|
# run's only number, and the log line above keeps the file
|
|
# pair beside it (the keep-final-values convention).
|
|
# Phase 129 (task 03): the chunk position is a sub-position
|
|
# of a file that's done — it clears in EVERY terminal, like
|
|
# success and failed (the convention split, pinned).
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image position clears in the
|
|
# CANCELLED terminal too — every terminal clears it (a
|
|
# sub-position of a file that's done/abandoned, the FILE
|
|
# counts above keep their final values).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|
|
# Phase 187 (task 02): the same two trailing fields as every
|
|
# other terminal — a cancelled run's memory is still data.
|
|
# Task 03 forces the sample first, so the cancelled payload
|
|
# keeps that same final pair (the status object stays
|
|
# authoritative after a cancel, as everywhere).
|
|
_sample_rss(force=True)
|
|
logger.info(
|
|
"sync: cancelled by user state=cancelled files_done=%d files_total=%d"
|
|
" rss_mb=%s rss_peak_mb=%s",
|
|
_status.files_done, _status.files_total,
|
|
fmt_mb(_status.rss_mb), fmt_mb(_status.rss_peak_mb),
|
|
)
|
|
raise
|
|
except Exception as e: # noqa: BLE001 — a background task dies in state, see above
|
|
logger.exception("sync: failed")
|
|
_status.state = "failed"
|
|
_status.finished_at = datetime.now(UTC)
|
|
_status.error = _sanitize_error(str(e))
|
|
# Phase 187 (task 02): ONE sanitized one-line summary beside the
|
|
# status field (the SAME masked string, so the log can never leak
|
|
# credentials the API does not) — the ``logger.exception`` above
|
|
# stays for the traceback. Task 03 forces the sample first, so
|
|
# the failed payload carries the same final pair as this line —
|
|
# a run that died mid-walk is exactly the run whose memory
|
|
# matters. _sample_rss never raises (app.core.memory never
|
|
# raises), so a memory read can never mask the failure.
|
|
_sample_rss(force=True)
|
|
logger.error(
|
|
"sync: failed state=failed files_done=%d files_total=%d"
|
|
" rss_mb=%s rss_peak_mb=%s error=%s",
|
|
_status.files_done, _status.files_total,
|
|
fmt_mb(_status.rss_mb), fmt_mb(_status.rss_peak_mb), _status.error,
|
|
)
|
|
_status.current_file = None # phase 64: keep the final counts
|
|
_status.phase = None # phase 98: keep the final summary counts
|
|
_status.current_summary = None
|
|
# Phase 194 (task 02): the scan position is kept here as well —
|
|
# a run that died mid-scan keeps the position it reached beside
|
|
# the final file counts (the convention, every terminal).
|
|
# Phase 129 (task 03): the chunk position clears in the
|
|
# failed terminal too (the convention split — the FILE counts
|
|
# keep their final values next to the error, the sub-position
|
|
# of a file that's done/abandoned does not).
|
|
_status.current_chunk = 0
|
|
_status.chunks_total = 0
|
|
# Phase 152 (task 01): the image position clears in the FAILED
|
|
# terminal too — the convention split (a sub-position of a
|
|
# file that's done/abandoned; the FILE counts keep their final
|
|
# values next to the error).
|
|
_status.images_done = 0
|
|
_status.images_total = 0
|