"""Sources sync API — one-click KB mirror (phase 32, task 01). Admin-only ``POST /api/sync`` + ``GET /api/sync/status`` behind the existing :func:`app.core.auth.require_admin` (A10 extended, phase 16 pattern — the public API surface stays stateless, the signed cookie remains the only session state, same as ``/api/steering``). The button's backend runs the full document sync **in-process** (A12 untouched — no queue, no new services): one ``asyncio`` background task plus a module-level :class:`SyncStatus` that the UI polls every 2 s (task 02). One sync at a time — ``POST`` while a run is in flight is 409; the status object is authoritative, so the UI can never sit on a stale button state (§7.4 adaptation, phase locked decisions). Phase 134 (task 01) adds the cancel: ``POST /api/sync/cancel`` (admin-only, same router) registers a COOPERATIVE cancel while a run is in flight (202, 409 ``no sync is running`` otherwise), and the run lands in the new ``cancelled`` TERMINAL state — a cancel is intent, not failure (``error = None``, ``detail = {}``, the final counts kept; the cancel paragraph below). Pipeline (the canonical "mirror the sources" action — phase locked decisions): 1. verify ``embed`` + summary model availability — fail fast before any clone (:func:`app.rag.llm.check_models`, phase 41): a dead model endpoint aborts the run naming the unavailable model, before source resolution or any ``clone_or_pull``; 2. resolve the effective sources — the ``git_sources`` DB rows (git **and** local, phase 38), else the ``BOR_GIT_SOURCES`` fallback (git-only) (:func:`app.rag.git_sources.effective_sources`, shared with the CLI) — empty on both origins (no git rows, no local rows, no env URLs) fails loudly (``no sources configured (git or local)``) instead of silently importing the legacy local directories; 3. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull` with the phase-121 clone URL (:func:`app.rag.git_sources.clone_url_for` — the row's ``token`` column injected as ``https://x-access-token:@…`` only for https? rows; NULL token → the bare stored URL verbatim, so public repos and legacy embedded-token rows clone exactly as before) and the row's BRANCH PIN as the third argument (phase 135 — a fresh clone checks it out, ``git clone --branch``; an existing checkout is guarded — a mismatched branch fails loudly, the remove-and-re-add error — and tracked, ``git fetch origin `` + ``git merge --ff-only FETCH_HEAD``; ``branch=None``, i.e. every pre-phase-135 row and every env-fallback row, keeps the pre-phase argv byte-identical) into ``BOR_SOURCES_DIR//`` (phase 28 — reused, not re-implemented; the checkout name stays on the bare URL — credential-free); ``kind=local`` → the stored directory, re-verified ``.is_dir()`` **at sync time** (it may have moved/deleted since add-time) — a missing directory raises ``local source missing: ``; a failing clone or a missing local dir aborts before any import; 4. ``import_sources(..., prune=True)`` over the single combined list (git checkouts + local dirs), honoring each row's ``ignore_paths`` (phase 89 — the per-root ignore map is built in the same per-row loop as the source list), its ``include_hidden`` flag (phase 105 — the per-root hidden-folders map, same per-row construction), and feeding each git checkout's per-file last-commit dates (phase 106, D2 — the per-root date map, ``file_commit_dates`` after the clone, same per-row construction; local rows contribute nothing and take the importer's mtime fallback) — prune so files deleted upstream, out of a local dir, or newly matching an ignore pattern leave the index (pruning covers the union; the CLI's no-prune default is unchanged); 5. when the import changed the KB (added + updated > 0), ``regenerate_overview`` refreshes the single ``kb_overview`` row (phase 31 trigger, best-effort inside) — and ``generate_ folder_summaries`` (phase 94, task 02) regenerates the stored folder summaries (the drill-down ``ls``'s per-level descriptions): its gate is the same change trigger — INCREMENTAL since phase 127 (task 03): the run's changed documents (``ImportSummary.changed_paths``, the import's added/updated/pruned ``(source, path)`` set) are passed as ``changed_docs=`` and ONLY their folder chains (each document's folder up to its source root — ``affected_folder_keys``) are rewritten, so a one-file change burns a handful of rows, not the whole table (the ``scripts/import_docs.py`` CLI keeps the byte-identical full regeneration — it passes no change set) **plus**, on an unchanged re-sync, a GAP probe — a candidate folder (at least 1 doc) with no stored row (``missing_folder_summaries``, phase 96, task 03): a gap fills ONLY the missing rows (``only_missing=True``; the old table-empty first-sync trigger is subsumed exactly — an empty table leaves every candidate missing), a complete table burns zero ``lite`` calls. It is per-folder fail-soft (a ``lite`` outage keeps the failed folders' previous rows — or leaves the row absent — and never flips the run to ``failed``) and only flushes — this run's own short-lived session commits (the phase-53 convention), so a folder failure never blocks step 6's bump; 6. when the import changed the KB (added + updated + pruned > 0 — the saved-chat invalidation gate, phase 53 task 02: a pruned document can invalidate a saved answer that cited it, deliberately broader than step 5's overview gate), the single-row ``sources_meta`` version counter is bumped exactly once in a short-lived session and the resulting generation lands in the status detail as ``sources_version`` (an unchanged re-sync reports the current generation without advancing it). A FAILED sync never bumps — the run aborts in the ``failed`` state before this step. Status is in memory: a restart mid-sync loses the running state (accepted — the next click re-syncs idempotently). The status also carries the phase-64 per-file progress — ``current_file`` (the ``source/relative/path`` the import is processing right now) plus ``files_done`` / ``files_total`` — null/0/0 before the import starts (clone/pull reports no file yet) and in terminal states, which clear ``current_file`` but keep the run's final counts. What the pair COUNTS changed in phase 194: on a progress caller (the UI sync is the only one) it is the position in the pre-scan's PREDICTED change set — ``files_total`` the number of files that will actually be re-indexed, ``files_done`` the 1-based position in that set — which is what makes the Sources page's ``N remaining`` say "10 remaining" on a ~10,000-file KB with 10 changes (the owner report of 2026-10-19) instead of counting the walk. ``BOR_SYNC_PENDING_COUNT=0`` restores the phase-193 walk position byte-for-byte; the terminal ``detail`` counts stay the exact surface either way. Phase 134 (task 01) makes the machine FIVE-state: ``cancelled`` joins ``success`` / ``failed`` as a TERMINAL state — it carries ``finished_at``, clears ``current_file`` / ``phase`` / ``current_summary`` and the chunk position, and keeps the run's final ``files_done`` / ``files_total`` + summary counts (the phase-64/98 keep-final-counts convention) — but with ``error = None`` and ``detail = {}``: a cancel is intent, not failure, so the UI never treats it as an error (no banner, no modal — a neutral result line). ``GET /api/sync/status`` gains nothing structurally — the ``state`` key simply reports ``"cancelled"``. The cancel is COOPERATIVE, best-effort (phase 134 design): while a run is in flight, ``POST /api/sync/cancel`` (the router's ``require_admin`` dependency applies — 403 anonymous) sets the module-level ``_cancel_event`` AND ``_task.cancel()`` (unblocking whatever ``await`` the pipeline is parked on — e.g. a per-file LLM call inside the import), answering 202 the moment the request is REGISTERED (the run may already be terminal by the next poll — the status object, not the task, is authoritative). ``start_sync`` clears the event when it starts a run, so a stale event from a previously cancelled run can never bleed into the next one. ``_run_sync`` checks the event at the pipeline boundaries via :func:`_check_cancelled` (raises ``asyncio.CancelledError`` when set): the top of the per-source loop (before each clone/pull or local-dir check), immediately before ``import_sources``, and immediately before the overview + summaries span (the ``added + updated > 0`` block). Phase 194 (task 02) adds one more await INSIDE the import span for the same shape: the pre-scan runs in a ``to_thread`` worker, so a cancel parked there unblocks the RUN at once (the ``cancelled`` terminal, nothing indexed) while the scan thread finishes the session of its own — the phase-134/179 split, the worker not being interruptible, pinned by ``tests/integration/test_sync_pending_count.py`` (importer side) and this module's cancel-during-scan pin. Consequence a future reader must know: a git subprocess already running at the cancel moment runs to completion (Python threads are not interruptible — since phase 179 it occupies a ``to_thread`` worker rather than the loop, so it no longer parks the EVENT LOOP while it does) — the cancel lands at the NEXT boundary (after that source's git ops, between files, or before overview/summaries); during the import span, the ``_task.cancel()`` from the endpoint interrupts at the next per-file await. Phase 179 changes only WHO waits: the run's await on the git worker is unblocked immediately by ``_task.cancel()`` (so the run reaches that next boundary) and every other request — the 2 s status poll included — is served while the git thread finishes on its own. The observable contract is phase 134's, unchanged: stop at the next boundary, ``cancelled`` terminal, no corruption. Phase 179 (task 02) — the sync's git operations run OFF the event loop. Before it, ``clone_or_pull`` and ``file_commit_dates`` were called as plain synchronous functions from inside ``_run_sync`` — the coroutine ``POST /api/sync`` spawns with ``asyncio.create_task`` — so their blocking ``subprocess.run`` (``scripts/git_sync.py``, ``run_git``) parked the app's SINGLE uvicorn loop thread for the whole git op: nothing else got scheduled, the ENTIRE HTTP surface stalled (the UI's 2 s ``GET /api/sync/status`` poll included) and the server resumed only when git returned. That is the owner's ``TODO.md`` L3 suspicion ("a long git clone seems to freeze the server?"), CONFIRMED by the in-process repro in ``tests/integration/test_sync_git_freeze.py`` — a 3 s stubbed clone made the status probe take ≈ 3.0 s (and ``GET /api/health`` too). Both git steps now go through ``asyncio.to_thread``, so the loop stays free for the whole clone while the clone/pull and the commit-date walk occupy a worker thread. Nothing else changes: the pipeline sequence, the error surfaces and the cancel terminals above are exactly as documented — only the SCHEDULING of the git calls moved (``scripts/git_sync.py`` itself is untouched, and it stays the module that owns git spawning — since phase 186 with exactly TWO sites there, the buffered ``run_git`` and the streaming ``stream_git_lines``, both reached only through ``to_thread``). Known BOUNDED non-fix (phase 179 decision A3): the run's short-lived ``SessionLocal()`` / ``effective_sources`` / version-bump calls stay ON the loop — millisecond-scale SQLAlchemy, not the reported symptom, deliberately out of scope. The ``CancelledError`` contract has TWO paths (the former "deliberately not caught" rule refined, not broken): in ``_run_sync``'s ``except asyncio.CancelledError`` clause (placed before the ``except Exception``), a USER-requested cancel (``_cancel_event`` set) lands the run in the ``cancelled`` terminal + one INFO line (PLAN §9: ``sync: cancelled by user state=cancelled files_done=… files_total=…``) and then RE-RAISES (the task ends cancelled — ``_task.done()`` is True, the status object stays authoritative); an UNREQUESTED cancel (the app-shutdown path — the event unset) re-raises UNCHANGED (the shutdown contract holds: a background task must die in state, never as an unobserved exception — swallowing an unrequested stop would mask a real one). Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the pipeline stage the run is in: the model-check + clone/pull prelude reports ``null`` (the bare "Syncing…" label stays), then ``"scan"`` (phase 194, task 02 — set immediately before ``import_sources``, REPLACING the pre-phase ``"import"`` set: what happens first in that call is the pre-scan that predicts the change set, and the label the task-03 branch renders for this phase is ``Scanning… (n/total)``), then ``"import"`` (flipped by the FIRST per-file ``progress`` call while the phase is still ``"scan"`` — the flip lives in the runner's own ``_hook`` closure, so the importer gains no extra callback; with ``BOR_SYNC_PENDING_COUNT=0`` no scan report fires at all and the flip is simply the first file), ``"overview"`` (set before ``regenerate_overview`` — a changed KB only), and ``"summaries"`` (set before a folder-summary generation — the changed-KB incremental regeneration (phase 127, task 03 — the changed set's folder keys only) or the unchanged-walk gap-fill; the no-gap skip sets NO phase and the run stays ``"import"``). While in ``"summaries"`` the generator's ``on_progress`` hook fills ``current_summary`` (the ``source`` / ``source/folder_path`` being summarized — the bare source name for the source-root row) plus ``summaries_done`` / ``summaries_total`` — the per-folder position through the long summary span where the file count sits still (the hook's ``total`` is the run's key count — the AFFECTED count on a changed KB, the missing count on a gap-fill). Terminal states (success, failed, and — phase 134 — cancelled) clear ``phase`` + ``current_summary`` but keep the run's final ``summaries_done`` / ``summaries_total`` (the phase-64 keep-final-counts convention). Phase 129 (task 03) adds the in-flight file's EMBED-chunk position — ``current_chunk`` (the 1-based first chunk of the in-flight token-budgeted batch) + ``chunks_total`` (the file's total content chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed` ``on_progress`` hook (tasks 01/02). They are 0/0 before the first batch (the prelude), during the ``"overview"`` / ``"summaries"`` phases, and in terminal states — the import phase clearing them at EVERY point it ends (the ``phase = "overview"`` and ``phase = "summaries"`` transitions and every terminal, the same spots ``current_file`` clears). Unlike the phase-64 FILE counts, the chunk position is a SUB-position of a file that's done — the keep-final- counts convention does NOT extend to it: the file counts survive the terminal for the result line's context, the chunk position clears with the file. Phase 152 (task 01) adds the in-flight file's VISION-image position — ``images_done`` (the 1-based index of the image about to be described and embedded) + ``images_total`` (the file's total images for this run — the three PDF vision spans of ONE file share ONE continuous counter per file; task 02 wires the importer's ``on_image_progress`` hook, task 03 the runner's closure that feeds these fields). They are 0/0 before the first image, during the ``"overview"`` / ``"summaries"`` phases, and in terminal states — the same spots ``current_chunk`` clears. Unlike the chunk position (every file embeds chunks), a file may have NO images at all, so the per-file ``progress`` hook ALSO resets them to 0/0: the pairing rule, extended — a ``progress`` call names the file and clears the image position, and every image position until the next ``progress`` call belongs to it (a file with no images must not inherit the previous PDF's position). ASSUMPTION A5: no new ``phase`` value — the image spans live inside ``phase: "import"``. Phase 187 (task 02) makes the run's MEMORY a logged number: every lifecycle line gains trailing RSS fields read through :mod:`app.core.memory` — ``sync: started … rss_mb=N`` and, in every terminal, ``… rss_mb=N rss_peak_mb=M`` (the ``import: summary …`` line carries ``rss_mb=`` too, on the importer side). APPEND-ONLY (LOCKED A2): no existing field is renamed, reordered or re-typed, so every pre-phase ``grep`` keeps working; an unknowable reading renders ``rss_mb=?`` through the ONE :func:`app.core.memory.fmt_mb` helper. The numbers are PROCESS numbers, not the host's cgroup figure — the :mod:`app.core.memory` module docstring owns that caveat. The ``failed`` funnel keeps its ``logger.exception`` (the traceback) and gains ONE sanitized one-line summary beside the status field, so the raw exception string never reaches the log un-sanitized. Phase 187 (task 03) puts the same numbers on the STATUS surface: the status object carries ``rss_mb`` / ``rss_peak_mb`` (``int | null``, sampled through the ONE :func:`_sample_rss` — at the run start, at every ``phase`` transition, in every terminal, and at most once per ``_RSS_SAMPLE_INTERVAL_S`` from the per-file progress hook, so a 27k-file walk costs ~600 ``/proc`` reads and not 27k — LOCKED A4), and ``GET /api/sync/status`` reports both keys ADDITIVELY. Phase 194 (task 02) extends both halves of that rule to the new stage: the ``"scan"`` set before ``import_sources`` and the ``scan``→``import`` flip are phase TRANSITIONS and FORCE the read, while the pre-scan's own per-file hook samples UNFORCED (throttled) — a 10,000-file scan costs one ``/proc`` read per interval, never one per file. The terminals keep the FINAL reading next to the final counts (the phase-64 convention), so the page that started the run answers the incident's question without a shell. A reading the platform cannot know stays ``null`` on the wire (the UI renders no clause for it) — never 0, never an invented number. The ``failed`` state's ``error`` string is masked by the shared sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors` (imported here under the private name ``_sanitize_error``). """ from __future__ import annotations import asyncio import logging import time from dataclasses import dataclass, field from datetime import UTC, datetime from pathlib import Path from typing import Any, Literal from fastapi import APIRouter, Depends, HTTPException from app.config import get_settings from app.core.auth import require_admin from app.core.errors import sanitize_error as _sanitize_error from app.core.memory import fmt_mb, rss_mb, rss_peak_mb from app.db import SessionLocal from app.rag.folder_summaries import generate_folder_summaries, missing_folder_summaries from app.rag.git_sources import clone_url_for, effective_sources from app.rag.importer import ImportSummary, import_sources from app.rag.llm import LLMClient, check_models from app.rag.overview import regenerate_overview from app.rag.sources_meta import bump_sources_version, current_sources_version from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates from scripts.import_docs import repo_name logger = logging.getLogger("app.api.sync") router = APIRouter( prefix="/sync", tags=["sync"], dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface ) @dataclass class SyncStatus: """In-memory state of the (at most one) in-flight sync run. ``state`` is a five-state machine: ``idle`` (never run / reset), ``running``, ``success``, ``failed``, and ``cancelled`` (phase 134, task 01 — terminal like ``success`` / ``failed``: ``finished_at`` set, ``current_file`` / ``phase`` / ``current_summary`` cleared, the final file + summary counts kept, but ``error = None`` and ``detail = {}`` — a cancel is intent, not failure). Terminal states carry the run's ``detail`` (success) or ``error`` (failure) so the UI can render the last result after a page reload (task 02's re-attach behavior). Phase 64 (task 02) progress fields: ``current_file`` is the ``source/relative/path`` the import is processing right now (null outside the import phase — clone/pull first, terminal states after); ``files_done`` / ``files_total`` carry the hook's done/total position and survive a terminal state (the run's last position is useful context next to the error). Since phase 194 the pair is the hook's PENDING position (phase 194: the pre-scan's predicted change set — ``files_total`` the files that will actually be re-indexed), not the walk position; ``BOR_SYNC_PENDING_COUNT=0`` restores the walk position and every other caller (the CLI, the upload job) is unaffected. Phase 194 (task 02) scan fields: ``scan_done`` / ``scan_total`` carry the pre-scan's position — the 1-based file whose hash+read just completed over the combined importable count the scan walks (the importer's ``on_scan_progress`` hook). They are 0/0 before the scan starts and on the ``BOR_SYNC_PENDING_COUNT=0`` rollback (no scan, so no report), reset with the run like every other field, and KEPT at every terminal beside the final file counts (the phase-64 keep-final-values convention — the scan the owner watched is still on the page after the run). ``current_file`` stays ``null`` through the phase: the scan names no file, it only counts. The degenerate empty knowledge base is the honest edge — the scan predicts nothing so no report fires and no ``progress`` call ever arrives: the phase stays ``"scan"`` until the next transition or the terminal (which clears it), and the label is the bare phase name with no count part (the UI's ``total === 0`` guard). Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage (null in the model-check + clone/pull prelude and in terminal states — the module docstring's phase machine, whose stages are now ``scan`` → ``import`` → ``overview`` → ``summaries``); while in the ``"summaries"`` phase, ``current_summary`` is the source / ``source/folder_path`` the generator is summarizing right now and ``summaries_done`` / ``summaries_total`` carry the generator's progress hook's position — the hook's ``total`` is the run's key count: the AFFECTED count on a changed KB (phase 127, task 03 — the changed set's folder keys, NOT the table's candidate count) or the missing count on an unchanged-walk gap-fill (as before). Terminal states clear ``phase`` + ``current_summary`` but keep the run's final summary counts (the phase-64 keep-final-counts convention). Phase 129 (task 03) chunk-position fields: ``current_chunk`` is the 1-based first chunk of the in-flight token-budgeted batch and ``chunks_total`` the in-flight file's total content chunks (batch granularity, the ``LLMClient.embed`` hook — the module docstring's contract). They are 0/0 before the first batch, during the ``"overview"`` / ``"summaries"`` phases, and in terminal states — the import phase clears them at every point it ends (the overview/summaries transitions + every terminal, where ``current_file`` also clears). The phase-64 keep-final-counts convention covers the FILE counts only — the chunk position is a sub-position of a file that's done, so it clears in the terminal while ``files_done`` / ``files_total`` keep their final values. Phase 152 (task 01) image-position fields: ``images_done`` is the 1-based index of the vision image about to be described and ``images_total`` the in-flight file's total images for this run (the importer's ``on_image_progress`` hook — the three PDF vision spans of one file share ONE continuous counter, the module docstring's contract). Like the chunk position, the image position is a SUB-position of a file that's done: the keep-final-counts convention covers the FILE counts only, so it clears with the file — 0/0 before the first image, during the ``"overview"`` / ``"summaries"`` phases, and in terminal states (the same spots the chunk position clears) — and the per-file ``progress`` hook resets it to 0/0 (a ``progress`` call names the file and clears the image position; a file with no images must not inherit the previous PDF's position). Phase 187 (task 03) memory fields: ``rss_mb`` is the process's CURRENT resident set size in MB and ``rss_peak_mb`` its HIGH-WATER mark — both PROCESS readings from :mod:`app.core.memory`, never the host's cgroup figure (that one counts page cache; the caveat lives in that module). They are written by ONE sampler, :func:`_sample_rss`: forced at the run start, at every ``phase`` transition and in every terminal, and throttled to one read per ``_RSS_SAMPLE_INTERVAL_S`` from the per-file ``progress`` hook (LOCKED A4 — never one ``/proc`` read per file). Both reset to ``None`` with the run — an idle or never-run status reports null, and the Sources page renders no peak clause for null — and every terminal KEEPS the final reading next to the final counts (the phase-64 keep-final-values convention, the one that makes "how much RAM did that sync use?" answerable from the page after the run, not only from the log). """ state: Literal["idle", "running", "success", "failed", "cancelled"] = "idle" started_at: datetime | None = None finished_at: datetime | None = None detail: dict[str, Any] = field(default_factory=dict) error: str | None = None # Phase 64 (task 02): per-file progress — the file the import is # processing right now and the hook's done/total position. current_file: str | None = None files_done: int = 0 files_total: int = 0 # Phase 194 (task 02): the PRE-SCAN's position — the 1-based file # whose hash+read just completed and the combined importable count # the scan walks (the importer's on_scan_progress hook). 0/0 before # the scan starts (and on the BOR_SYNC_PENDING_COUNT=0 rollback); # reset with the run and kept at every terminal beside the final # file counts (the phase-64 keep-final-values convention). scan_done: int = 0 scan_total: int = 0 # Phase 98 (task 01): the phase machine — the pipeline stage the # run is in and, while in the folder-summary phase, the folder # being summarized plus the hook's done/total position. Phase 194 # (task 02) adds the "scan" stage (the pre-scan that predicts the # run's change set — the module docstring's phase machine). phase: Literal["scan", "import", "overview", "summaries"] | None = None current_summary: str | None = None summaries_done: int = 0 summaries_total: int = 0 # Phase 129 (task 03): the current file's EMBED chunk position — # the first chunk of the in-flight token-budgeted batch and the # file's total content chunks (the LLMClient.embed hook, batch # granularity). 0/0 before the first batch, during the # prelude/overview/summaries phases, and in terminal states. current_chunk: int = 0 chunks_total: int = 0 # Phase 152 (task 01): the in-flight file's VISION-image position — # the 1-based index of the image about to be described and the # file's total images for this run (the importer's # on_image_progress hook; the three PDF vision spans share ONE # continuous counter per file). 0/0 before the first image, during # the prelude/overview/summaries phases, and in terminal states. images_done: int = 0 images_total: int = 0 # Phase 187 (task 03): the run's PROCESS memory in MB — current and # high-water (app.core.memory, never the host's cgroup figure). None # idle / never-run and reset with the run; kept as the FINAL reading # in every terminal. Written ONLY through _sample_rss, the one # sampling point (LOCKED A4). rss_mb: int | None = None rss_peak_mb: int | None = None _status = SyncStatus() _task: asyncio.Task[None] | None = None # Phase 134 (task 01): the cancel request — set by POST # /api/sync/cancel while a run is in flight; _run_sync's boundaries # + its ``except CancelledError`` clause read it (the module # docstring's cancel paragraph). Cleared by start_sync when a run # starts, so a stale event can never bleed into the next run. _cancel_event = asyncio.Event() #: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hooks may #: re-read the process memory, in seconds. Those hooks fire once per #: file — the import's ``progress`` AND, since phase 194 (task 02), the #: pre-scan's ``on_scan_progress`` — so without this guard a 27k-file #: walk would cost 27k ``/proc/self/status`` reads; with it the same walk #: costs ~600 and the operator watching the page still sees the number #: move (PLAN §7.4's never-stale contract). Every other sampling point #: (the run start, the ``phase`` transitions including the ``scan`` set #: and its flip to ``import``, the terminals) FORCES the read instead. _RSS_SAMPLE_INTERVAL_S = 5.0 #: ``time.monotonic()`` of the last RSS read; ``-inf`` means "never", so #: the first throttled call reads instead of waiting out the interval. #: Module-global like ``_status`` itself (the tests' fresh-state fixture #: re-seeds both). _rss_sampled_at: float = float("-inf") def _sample_rss(force: bool = False) -> None: """Read the process memory onto ``_status`` — the ONE sampling point (phase 187, task 03). ``force=True`` reads NOW: the run start, each ``phase`` transition, every terminal — the points where the number is worth a syscall and where a stale reading would be a lie. ``force=False`` (the per-file progress hook) is a NO-OP inside :data:`_RSS_SAMPLE_INTERVAL_S` of the previous read, which is what makes the sampling throttled rather than per-file (LOCKED A4). Both fields come off the SAME read, so the pair the status endpoint and the lifecycle line report is one instant, not two. Nothing here raises: :mod:`app.core.memory` degrades a platform that cannot know to ``None``, which the status renders as JSON null (no UI clause) and the log as ``?``. **The pair is normalized to one truth: ``peak >= current``.** The two numbers come from two different kernel counters read at two instants (``VmRSS`` from ``/proc/self/status``, then ``ru_maxrss``), and the kernel refreshes its high-water LAZILY, so a growing process really can report a current reading ABOVE the reported high-water (measured on this repo's own suite: ``rss_mb=164`` against ``rss_peak_mb=162``). A "peak" below the run's own current footprint is not a smaller number, it is a FALSE one — the high-water is by definition at least the current set — so the ONE sampling point raises it to the current reading. That is a lower bound on the truth, never an invention, and it is why every consumer (both lifecycle lines, the status payload, the page's clause) can now take ``peak >= current`` as an invariant instead of carrying a slack. A ``None`` is still a ``None``: an unknowable peak is never fabricated from the current reading. """ global _rss_sampled_at now = time.monotonic() if not force and (now - _rss_sampled_at) < _RSS_SAMPLE_INTERVAL_S: return _rss_sampled_at = now current = rss_mb() peak = rss_peak_mb() if current is not None and peak is not None and peak < current: peak = current # a high-water below the current set is a stale one _status.rss_mb = current _status.rss_peak_mb = peak @router.get("/status") def sync_status() -> dict[str, Any]: """Current sync state (the UI polls this every 2 s — task 02). ``started_at`` / ``finished_at`` are ISO-8601 strings or null. ``current_file`` (phase 64) is the ``source/relative/path`` the import is processing right now — null during the clone/pull phase and in terminal states; ``files_done`` / ``files_total`` carry the hook's position (0/0 idle) and keep their final values in the terminal — the position in the pre-scan's PREDICTED change set on a progress caller (phase 194), the walk position under ``BOR_SYNC_PENDING_COUNT=0``. ``scan_done`` / ``scan_total`` (phase 194) are the PRE-SCAN's position: the 1-based file whose hash+compare just ran over the combined importable count the scan walks. They are 0/0 before the scan starts — the prelude, a run on the ``0`` rollback, and every run that never happened — and in a terminal they keep the FINAL scan values beside the final file counts (the phase-64 keep-final-values convention); ``current_file`` stays null while ``phase`` is ``"scan"``, because the scan names no file. Both keys are ADDITIVE — placed after ``files_total``, every pre-existing key keeping its name, type and position. The degenerate empty knowledge base reports the phase without ever reporting a position (0/0 — the UI's ``total === 0`` guard renders no count part). ``current_chunk`` / ``chunks_total`` (phase 129) are the in-flight file's embed-chunk position (batch granularity — the first chunk of the in-flight batch over the file's total content chunks): 0/0 before the first batch, in the prelude/overview/ summaries phases, and in terminal states (they clear at every point the import phase ends; the FILE counts do not). ``phase`` (phase 98) is the pipeline stage — null in the prelude and terminal states; while in ``"summaries"``, ``current_summary`` + ``summaries_done`` / ``summaries_total`` carry the generator's position (the terminal keeps the final summary counts). Phase 134 (task 01): ``state`` also reports ``"cancelled"`` for a user-cancelled run — a terminal state with ``error`` null and ``detail`` empty; the run's final file + summary counts are kept, the chunk position cleared. Phase 152 (task 01): ``images_done`` / ``images_total`` are the in-flight file's vision-image position (the importer's ``on_image_progress`` hook — the three PDF vision spans of one file share ONE continuous counter): 0/0 before the first image, in the prelude/overview/summaries phases, and in terminal states; the per-file ``progress`` hook clears it at every file boundary (a file with no images never shows a stale position). Phase 187 (task 03): ``rss_mb`` / ``rss_peak_mb`` are the run's PROCESS memory in MB — the current resident set and the high-water mark (:mod:`app.core.memory`, never the host's cgroup figure, which counts page cache). ``null`` means "no reading", which is the idle / never-run state AND any platform that cannot know — never 0, and never an invented number. The run samples them through :func:`_sample_rss`: at the run start, at every ``phase`` transition, throttled to one read per 5 s through the import's per-file hook, and FORCED in every terminal — so a terminal reports the run's FINAL reading next to its final counts (the phase-64 convention, same as ``files_done`` / ``files_total``), and a reload after the run still shows the number that ``sync: done …`` logged. Additive only — every pre-existing key keeps its name, type, and meaning. """ return { "state": _status.state, "started_at": _status.started_at.isoformat() if _status.started_at else None, "finished_at": _status.finished_at.isoformat() if _status.finished_at else None, "detail": _status.detail, "error": _status.error, "current_file": _status.current_file, "files_done": _status.files_done, "files_total": _status.files_total, # Phase 194 (task 02): the pre-scan's position (the importer's # on_scan_progress hook) — 0/0 before the scan starts and on the # BOR_SYNC_PENDING_COUNT=0 rollback, the FINAL pair kept in every # terminal beside the file counts above (the keep-final-values # convention). Additive: every pre-existing key keeps its name, # type and position. "scan_done": _status.scan_done, "scan_total": _status.scan_total, # Phase 129 (task 03): the in-flight file's embed-chunk # position — the dataclass defaults make idle 0/0, and the # runner clears it at every point the import phase ends # (overview/summaries transitions + every terminal). "current_chunk": _status.current_chunk, "chunks_total": _status.chunks_total, # Phase 152 (task 01): the in-flight file's VISION-image # position (the importer's on_image_progress hook — the three # PDF vision spans share ONE continuous counter per file): # 0/0 before the first image, during the # prelude/overview/summaries phases, and in terminal states # (cleared at every point the import phase ends; the per-file # progress hook re-clears it at every file boundary). "images_done": _status.images_done, "images_total": _status.images_total, # Phase 98 (task 01): the phase machine — null/0/0 idle (the # dataclass defaults) and in terminal states (which clear # phase + current_summary but keep the final summary counts). "phase": _status.phase, "current_summary": _status.current_summary, "summaries_done": _status.summaries_done, "summaries_total": _status.summaries_total, # Phase 187 (task 03): the run's PROCESS memory in MB (current + # high-water), read through the ONE _sample_rss. null idle / # never-run; in a terminal it is the run's FINAL reading (the # phase-64 keep-final-values convention — the number the # 2026-10-15 incident had to reconstruct from systemd teardown # lines, now on the page that started the run). "rss_mb": _status.rss_mb, "rss_peak_mb": _status.rss_peak_mb, } @router.post("", status_code=202) async def start_sync() -> dict[str, str]: """Start the clone → import → overview sync as a background task. 202 + ``sync started`` kicks off :func:`_run_sync` on the app's event loop. 409 when a run is already in flight (one sync at a time — the status endpoint is the single source of truth for the run, and the UI re-attaches to it rather than starting a second one). Phase 134 (task 01): the run's cancel request resets with the run — ``_cancel_event.clear()`` before the task is created, so a stale event from a previously cancelled run can never bleed into this one. """ global _task if _task is not None and not _task.done(): raise HTTPException(status_code=409, detail="a sync is already running") _cancel_event.clear() _task = asyncio.create_task(_run_sync()) return {"detail": "sync started"} @router.post("/cancel", status_code=202) async def cancel_sync() -> dict[str, str]: """Request the cancel of the in-flight run (phase 134, task 01). 202 + ``cancel requested`` is answered the moment the request is REGISTERED: the module ``_cancel_event`` is set (the runner's ``_check_cancelled`` boundaries + its ``except CancelledError`` conversion see it) AND ``_task.cancel()`` unblocks whatever ``await`` the pipeline is parked on (e.g. a per-file LLM call inside the import). The run may already be terminal by the next status poll — the status object, not the task, is authoritative, so the UI's poll settles on the terminal (``cancelled``) state. 409 ``no sync is running`` when there is no run in flight (the ``_task`` is None or done — the harmless race the UI's poll settles; no error copy, no state change). """ if _task is None or _task.done(): raise HTTPException(status_code=409, detail="no sync is running") _cancel_event.set() _task.cancel() return {"detail": "cancel requested"} def _check_cancelled() -> None: """Phase 134 (task 01): the cooperative cancel boundary. Raises ``asyncio.CancelledError`` when the module ``_cancel_event`` is set (the user requested a stop — ``POST /api/sync/cancel``); :func:`_run_sync`'s ``except asyncio.CancelledError`` clause converts it into the ``cancelled`` terminal state. Called at the pipeline boundaries (the module docstring's cancel paragraph) — the top of the per-source loop, before ``import_sources``, and before the overview + summaries span. A git subprocess in flight at the cancel moment runs to completion in its ``to_thread`` worker (phase 179: threads are not interruptible, but the loop is no longer parked by it); the cancel lands at the NEXT boundary — or, when ``_task.cancel()`` interrupts the ``to_thread`` await itself, right there, with the worker finishing on its own. """ if _cancel_event.is_set(): raise asyncio.CancelledError() async def _run_sync() -> None: """The full sync pipeline, one in-process background task. Every failure mode (git, embeddings, anything else) lands in the ``failed`` state with a sanitized ``error`` string — a background task must die in state, never as an unobserved exception. Cancellation (phase 134, task 01 — the module docstring's cancel paragraph): the cooperative boundaries (:func:`_check_cancelled` at the top of the per-source loop, before ``import_sources``, and before the overview + summaries span) plus the ``except asyncio.CancelledError`` clause convert a USER-requested cancel (``_cancel_event`` set by ``POST /api/sync/cancel``) into the ``cancelled`` terminal state + one INFO line, then re-raise (the task ends cancelled — the status object stays authoritative). An UNREQUESTED cancel (the app-shutdown path — the event unset) re-raises unchanged: a background task is never swallowed, and swallowing a real stop would mask it. Phase 179 (task 02): the per-source git steps (``clone_or_pull``, ``file_commit_dates``) run off the loop via ``asyncio.to_thread`` so a long clone no longer stalls the rest of the HTTP surface (the module docstring's freeze paragraph); the run's own sequencing is byte-for-byte the pipeline above. Phase 194 (task 02): the phase set before ``import_sources`` is ``"scan"`` (not ``"import"``), because that call begins with the off-loop pre-scan that predicts the change set; the FIRST per-file ``progress`` call flips it to ``"import"`` inside ``_hook``. Both transitions force the memory sample, the scan's own per-file hook does not (LOCKED A4), and ``scan_done`` / ``scan_total`` reset with the run and survive every terminal beside the file counts. """ _status.state = "running" _status.started_at = datetime.now(UTC) _status.finished_at = None _status.detail = {} _status.error = None # Phase 64 (task 02): the progress fields reset with the run — no # current file until the import starts (the clone/pull phase). _status.current_file = None _status.files_done = 0 _status.files_total = 0 # Phase 194 (task 02): the pre-scan's position resets with the run — # 0/0 until the scan reports its first file (and 0/0 for the whole # run on the BOR_SYNC_PENDING_COUNT=0 rollback, which scans nothing). _status.scan_done = 0 _status.scan_total = 0 # Phase 98 (task 01): the phase fields reset with the run — no # phase until the import starts (the model-check + clone/pull # prelude reports null), no folder until the summary span starts. _status.phase = None _status.current_summary = None _status.summaries_done = 0 _status.summaries_total = 0 # Phase 129 (task 03): the chunk-position fields reset with the # run — 0/0 until the import's first embed batch reports. _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image-position fields reset with the # run — 0/0 until the import's first vision span reports (the # per-file progress hook re-clears them at every file boundary). _status.images_done = 0 _status.images_total = 0 # Phase 187 (task 03): the run-start memory sample — forced, so it # BOTH replaces the previous run's readings (a forced sample fully # determines the pair; None where the platform cannot know) and # re-arms the 5 s throttle. No stale peak ever bleeds into a new # run's status. _sample_rss(force=True) try: settings = get_settings() # Step 1 (phase 41): fail fast — verify both models the sync # needs (embed + summary) before source resolution or any # clone. The client is reused for the import + overview below. llm = LLMClient() await check_models(llm) # The background task has no request session: open a short-lived # one around the shared phase-35/38 resolver (DB rows of both # kinds win; the BOR_GIT_SOURCES git list is a fallback while # the table is empty). db = SessionLocal() try: rows, origin = effective_sources(db) finally: db.close() if not rows: # The button targets the admin-managed source registry # (manual --source dirs have no repo to clone) — an empty # config on *both* origins (no git rows, no local rows, no # env URLs) fails loudly instead of silently importing the # legacy directories. raise GitSyncError("no sources configured (git or local)") git_count = sum(1 for row in rows if row.kind == "git") # Phase 187 (task 02): the trailing rss_mb= is APPEND-ONLY (LOCKED # A2) — every pre-phase grep of this line keeps matching. Task 03 # logs the run-start SAMPLE (the reading the status reports) so # the line and the payload never disagree — the shape is # byte-identical to task 02's. logger.info( "sync: started repos=%d origin=%s git=%d local=%d rss_mb=%s", len(rows), origin, git_count, len(rows) - git_count, fmt_mb(_status.rss_mb), ) sources_root = Path(settings.sources_dir).expanduser() sources: list[Path] = [] ignore_by_root: dict[str, list[str]] = {} include_hidden_by_root: dict[str, bool] = {} doc_dates_by_root: dict[str, dict[str, datetime]] = {} for row in rows: # Phase 134 (task 01): the per-source boundary — a cancel # registered while the previous source's git ops run lands # here, before this source's clone/pull or local-dir check. # Phase 179 (task 02): those git ops now run in a # ``to_thread`` worker, so a cancel parked on them no # longer means a PARKED EVENT LOOP — ``_task.cancel()`` # unblocks this run at the ``await`` and the loop serves # everything else immediately, while the in-flight git # thread runs its subprocess to completion (Python threads # are not interruptible). Contract unchanged (phase 134): # stop at the next boundary, ``cancelled`` terminal. _check_cancelled() if row.kind == "git": # Phase 121: the token column is injected into the clone # URL ONLY here (clone_url_for — NULL token → the bare # stored URL verbatim); repo_name stays on the bare URL # so the checkout directory name is credential-free. # Phase 135: the row's branch pin is the third # argument — NULL (every pre-phase-135 row + every # env-fallback row) keeps the pre-phase argv # byte-identical; a pinned branch is checked out on a # fresh clone (--branch) and guarded + tracked on an # existing checkout (fetch + ff-merge of that ref). # Phase 179 (task 02): OFF the event loop — the # blocking subprocess inside clone_or_pull used to # stall the whole HTTP surface for the duration of the # clone, the UI's 2 s GET /api/sync/status poll first # among the victims; to_thread keeps the loop serving # while the git worker runs. root = await asyncio.to_thread( clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch, ) # Phase 106 (D2): the checkout's per-file last-commit # dates, keyed by the SAME root string the importer # sees (full-history checkouts → true per-file # last-commit dates for every git source — phase 107 # D11 supersedes phase 106 D10's shallow tip-date # behavior). Local rows contribute nothing — the # importer's mtime fallback applies to them. # Phase 179 (task 02): off the loop for the same # reason — a full-history ``git log`` walk over a big # checkout is as blocking as the clone was. doc_dates_by_root[str(root)] = await asyncio.to_thread( file_commit_dates, root ) else: # kind=local — the stored expanded path (phase 38 also # mirrors it in the NOT-NULL ``url`` location column, the # ``or`` keeps the type checker honest); re-verified at # sync time because the directory may have moved or been # deleted since add-time. root = Path(row.path or row.url).expanduser() if not root.is_dir(): raise GitSyncError(f"local source missing: {root}") sources.append(root) # Phase 89: the row's ignore list, keyed by the SAME root # string the importer sees; two rows sharing a root string # get the union (extend, not replace) — the sibling/repo-name # edge. if row.ignore_paths: ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths) # Phase 105 (A1/A4): the row's hidden-folders flag, keyed by # the SAME root string the importer sees; a shared-root # collision ORs — if EITHER row says "index hidden", the # root does (the ignore-map union's boolean mirror). include_hidden_by_root[str(root)] = ( include_hidden_by_root.get(str(root), False) or bool(row.include_hidden) ) # Phase 64 (task 02): the per-file progress hook — the status # endpoint reports the file being processed right now. The # closure captures the module ``_status`` exactly like the state # assignments above. def _hook(source: str, rel: str, done: int, total: int) -> None: _status.current_file = f"{source}/{rel}" _status.files_done = done _status.files_total = total # Phase 152 (task 01): the pairing rule, extended — a # progress call names the file AND clears the image # position: every image position until the next progress # call belongs to this file. A file with no images must # not inherit the previous PDF's position (unlike the # chunk position, not every file embeds chunks, so the # run-start reset alone cannot cover it). _status.images_done = 0 _status.images_total = 0 # Phase 194 (task 02): the scan→import FLIP. The phase is # set to "scan" immediately before import_sources, and the # first file the walk actually reaches is where the running # label stops being a scan — no extra callback reaches the # importer, the flip lives in this existing per-file hook. if _status.phase == "scan": _status.phase = "import" # A phase TRANSITION is a forced sampling point (phase # 187), so this one read is not the throttled per-file # one the tail of this closure does. _sample_rss(force=True) # Phase 187 (task 03): the run's memory, sampled THROTTLED # (unforced) — this hook fires once per file, so a 27k-file # walk reads /proc ~600 times instead of 27k (LOCKED A4) # while the number on the page still climbs (PLAN §7.4). _sample_rss() # Phase 129 (task 03): the in-flight file's embed-chunk # position — the LLMClient.embed hook (task 01) fires before # each token-budgeted batch with (first_chunk_1based, total); # the file's IDENTITY is the sibling ``progress`` hook's (the # two hooks are paired — a progress call names the file, and # every chunk position until the next one belongs to it). def _chunk_hook(chunk: int, total: int) -> None: _status.current_chunk = chunk _status.chunks_total = total # Phase 152 (task 03): the in-flight file's VISION-image # position — the importer's ``on_image_progress`` hook (task # 02) fires before each of the PDF's vision describe calls # with (position_1based, total), the file's ONE continuous # counter across its three vision spans (the importer owns # the continuity — this closure only lands the pair); the # file's IDENTITY is the sibling ``progress`` hook's, which # ALSO resets this position (the three hooks are paired: # ``progress`` names the file and clears the image position, # ``on_chunk_progress`` reports the embed-batch position, # ``on_image_progress`` reports the vision-image position). def _image_hook(done: int, total: int) -> None: _status.images_done = done _status.images_total = total # Phase 194 (task 02): the PRE-SCAN's position — the importer's # ``on_scan_progress`` hook, fired once per importable file while # the run's change set is still being PREDICTED (before the walk), # with (position_1based, scan_total). It runs on the scan's # WORKER thread, so this closure only assigns fields on the # module's own status object — the house hook contract that makes # each call a single GIL-atomic store (phase 179). The memory # sample is UNFORCED here: this fires once per file, exactly like # ``_hook`` above, so it costs one /proc read per interval and # never one per file (LOCKED A4). def _scan_hook(scan: int, total: int) -> None: _status.scan_done = scan _status.scan_total = total _sample_rss() # Phase 98 (task 01) + phase 194 (task 02): the phase machine — # the SCAN phase starts NOW (the model-check + clone/pull prelude # above reported ``phase: null``). What runs first inside # ``import_sources`` is the pre-scan that predicts the change set, # and the label the owner reads through it is "Scanning…" # (phase 193's "import" set moved to the first ``_hook`` call # above — the flip), so the count the page shows is never a scan # wearing the work's shape. _status.phase = "scan" # Phase 187 (task 03): the phase transitions are the forced # sampling points — each stage of the run begins with its own # reading on the status (the per-file hook throttles). _sample_rss(force=True) # Phase 134 (task 01): the import boundary — a cancel between # the source walk and the import (or parked in a blocking git # span that just finished) lands here, before any file is # imported. _check_cancelled() summary: ImportSummary = await import_sources( sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook, on_image_progress=_image_hook, # phase 152 (task 03) on_scan_progress=_scan_hook, # phase 194 (task 02) ignore_by_root=ignore_by_root, include_hidden_by_root=include_hidden_by_root, doc_dates_by_root=doc_dates_by_root, ) # Phase 134 (task 01): the overview + summaries boundary — a # cancel during the import span (the endpoint's _task.cancel() # interrupts at the next per-file await) or registered anywhere # before this point lands here, before the lite spans start. _check_cancelled() overview = False if summary.added + summary.updated > 0: _status.phase = "overview" # phase 98 (task 01) _sample_rss(force=True) # phase 187 (task 03) # Phase 129 (task 03): the import phase ends here — the # chunk position (a sub-position of a done file) clears # with the phase change; the file counts keep their final # values (the phase-64 convention covers files only). _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image position clears at the # same point (a sub-position of a done file, like the # chunk position). _status.images_done = 0 _status.images_total = 0 overview = await regenerate_overview(llm) # Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03): # the folder summaries — the drill-down ls's per-level # descriptions. A changed KB (added + updated > 0) is an # INCREMENTAL regeneration (phase 127, task 03): the run's # changed documents (summary.changed_paths — the import's # added/updated/pruned set) are passed as changed_docs= and # ONLY their folder chains (each document's folder up to its # source root) are rewritten — a one-file change burns a # handful of rows, not the whole table (the CLI keeps the # byte-identical full regeneration — it passes no change set); # an unchanged re-sync takes the GAP probe instead of the old # table-empty one — a candidate folder (at least 1 doc — every # existing folder) with no stored row: a gap # fills ONLY the missing rows (only_missing=True — the # subsumed table-empty first-sync trigger included, where # every candidate is missing), a complete table burns zero # lite calls. The generator is per-folder fail-soft (a lite # outage never flips the run to failed) and only flushes: this # run's own short-lived session commits (the phase-53 # convention), so the step-6 bump stays change-gated on the KB, # not on the summaries. The status surface is the PHASE # machine (phase 98, task 01): the stats dict itself stays # log-only (the detail shape is untouched). # Phase 98 (task 01): the folder-summary progress hook — the # closure-captures-``_status`` convention the file's ``_hook`` # above already uses; the generator's ``total`` (the loop-start # key count — the affected count on a changed KB, the missing # count on a gap-fill) and the bare source name for the root # row come straight from the hook's arguments (D5's shape). def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None: _status.current_summary = ( source if folder_path == "" else f"{source}/{folder_path}" ) _status.summaries_done = done _status.summaries_total = total fs_db = SessionLocal() try: if summary.added + summary.updated > 0: # Changed KB: incremental regeneration (phase 127, # task 03) — ONLY the changed set's folder keys # (changed_docs=summary.changed_paths: each document's # folder chain up to its source root); the summaries # phase starts now, and the hook reports the per-folder # position through the (long) span (its total is the # affected count). _status.phase = "summaries" _sample_rss(force=True) # phase 187 (task 03) # Phase 129 (task 03): the import phase ends here too # (the changed-KB branch skips no phase) — same # chunk-position clear as the overview transition. _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image position clears at # the same point (a sub-position of a done file). _status.images_done = 0 _status.images_total = 0 folder_stats = await generate_folder_summaries( fs_db, llm, changed_docs=summary.changed_paths, on_progress=_summary_hook, ) fs_db.commit() logger.info( "sync: folder_summaries changed=%d stats=%s", len(summary.changed_paths), folder_stats, ) else: missing = missing_folder_summaries(fs_db) if missing: # Unchanged-walk gap-fill: the SAME summaries # phase + hook (the fill's total is the missing # count — the hook's loop-start ``total``). _status.phase = "summaries" _sample_rss(force=True) # phase 187 (task 03) # Phase 129 (task 03): the import phase ends here # too (the unchanged-walk gap-fill branch) — same # chunk-position clear. _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image position clears # at the same point (a sub-position of a done # file). _status.images_done = 0 _status.images_total = 0 folder_stats = await generate_folder_summaries( fs_db, llm, only_missing=True, on_progress=_summary_hook ) fs_db.commit() logger.info( "sync: folder_summaries gap-fill missing=%d stats=%s", len(missing), folder_stats, ) else: # The no-gap skip sets NO phase — the run stays # "import" through to the terminal (the phase-64 # bare-label pin holds for this shape). logger.info("sync: folder_summaries skipped (KB unchanged)") finally: fs_db.close() # Phase 53 (task 02): a sync that changed the KB advances the # sources version exactly once — the saved-chat invalidation # marker (task 03 stamps rows against it). The gate is # deliberately broader than the overview's above: a pruned # document can invalidate a saved answer that cited it, so # ``pruned > 0`` bumps too. The bump commits in its own short # session (the ``effective_sources`` pattern above), so it # lands even if the best-effort overview then fails — the index # really did change. An unchanged re-sync never bumps; it # reports the current generation instead, so the detail always # carries the generation the KB is now at. db = SessionLocal() try: if summary.added + summary.updated + summary.pruned > 0: sources_version = bump_sources_version(db) db.commit() else: sources_version = current_sources_version(db) finally: db.close() _status.state = "success" _status.finished_at = datetime.now(UTC) _status.current_file = None # phase 64: keep the final counts _status.phase = None # phase 98: keep the final summary counts _status.current_summary = None # Phase 194 (task 02): the scan position is NOT cleared here — # like the file counts it keeps its final values beside them # (the keep-final-values convention), so the page reloaded after # the run still says how big the predicted change set was. # Phase 129 (task 03): the chunk position is a sub-position of # a file that's done — it clears in the terminal (unlike the # FILE counts, which keep their final values for the result # line's context). _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image position is a sub-position of # a file that's done — it clears in the terminal too (the FILE # counts keep their final values, the sub-position does not). _status.images_done = 0 _status.images_total = 0 _status.detail = { "files": summary.files, "added": summary.added, "updated": summary.updated, "unchanged": summary.unchanged, "pruned": summary.pruned, "errors": summary.errors, "chunks": summary.chunks, "summaries": summary.summaries, "summary_errors": summary.summary_errors, "dates_updated": summary.dates_updated, "overview": overview, "sources_version": sources_version, } # Phase 187 (task 02): the memory the run is at NOW and its # high-water mark, next to the final counts (the phase-64 # keep-final-values convention, extended to the numbers the # 2026-10-15 incident had to reconstruct from systemd teardown # lines). Appended after ``detail=`` — the line stays greppable # as ``sync: done detail=``. Task 03 forces the sample FIRST and # logs it, so the payload's rss_mb / rss_peak_mb and this line # carry the SAME final pair (the phase-64 convention, kept # honest across both surfaces). _sample_rss(force=True) logger.info( "sync: done detail=%s rss_mb=%s rss_peak_mb=%s", _status.detail, fmt_mb(_status.rss_mb), fmt_mb(_status.rss_peak_mb), ) except asyncio.CancelledError: # Phase 134 (task 01): the TWO conversion paths (the module # docstring's cancel paragraph). A user-requested cancel — # ``_cancel_event`` set by POST /api/sync/cancel, whether the # endpoint's ``_task.cancel()`` interrupted a parked await or a # ``_check_cancelled`` boundary raised — lands the run in the # ``cancelled`` terminal and re-raises (the task ends # cancelled — ``_task.done()`` is True, the status object stays # authoritative). An unrequested cancel (the app-shutdown path # — the event unset) re-raises unchanged: never swallow a real # stop. if _cancel_event.is_set(): _status.state = "cancelled" _status.finished_at = datetime.now(UTC) _status.error = None # a cancel is intent, not failure _status.current_file = None # keep the final counts _status.phase = None # keep the final summary counts _status.current_summary = None # Phase 194 (task 02): the scan position survives the cancel # too — where a cancel landed DURING the pre-scan it is the # run's only number, and the log line above keeps the file # pair beside it (the keep-final-values convention). # Phase 129 (task 03): the chunk position is a sub-position # of a file that's done — it clears in EVERY terminal, like # success and failed (the convention split, pinned). _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image position clears in the # CANCELLED terminal too — every terminal clears it (a # sub-position of a file that's done/abandoned, the FILE # counts above keep their final values). _status.images_done = 0 _status.images_total = 0 # Phase 187 (task 02): the same two trailing fields as every # other terminal — a cancelled run's memory is still data. # Task 03 forces the sample first, so the cancelled payload # keeps that same final pair (the status object stays # authoritative after a cancel, as everywhere). _sample_rss(force=True) logger.info( "sync: cancelled by user state=cancelled files_done=%d files_total=%d" " rss_mb=%s rss_peak_mb=%s", _status.files_done, _status.files_total, fmt_mb(_status.rss_mb), fmt_mb(_status.rss_peak_mb), ) raise except Exception as e: # noqa: BLE001 — a background task dies in state, see above logger.exception("sync: failed") _status.state = "failed" _status.finished_at = datetime.now(UTC) _status.error = _sanitize_error(str(e)) # Phase 187 (task 02): ONE sanitized one-line summary beside the # status field (the SAME masked string, so the log can never leak # credentials the API does not) — the ``logger.exception`` above # stays for the traceback. Task 03 forces the sample first, so # the failed payload carries the same final pair as this line — # a run that died mid-walk is exactly the run whose memory # matters. _sample_rss never raises (app.core.memory never # raises), so a memory read can never mask the failure. _sample_rss(force=True) logger.error( "sync: failed state=failed files_done=%d files_total=%d" " rss_mb=%s rss_peak_mb=%s error=%s", _status.files_done, _status.files_total, fmt_mb(_status.rss_mb), fmt_mb(_status.rss_peak_mb), _status.error, ) _status.current_file = None # phase 64: keep the final counts _status.phase = None # phase 98: keep the final summary counts _status.current_summary = None # Phase 194 (task 02): the scan position is kept here as well — # a run that died mid-scan keeps the position it reached beside # the final file counts (the convention, every terminal). # Phase 129 (task 03): the chunk position clears in the # failed terminal too (the convention split — the FILE counts # keep their final values next to the error, the sub-position # of a file that's done/abandoned does not). _status.current_chunk = 0 _status.chunks_total = 0 # Phase 152 (task 01): the image position clears in the FAILED # terminal too — the convention split (a sub-position of a # file that's done/abandoned; the FILE counts keep their final # values next to the error). _status.images_done = 0 _status.images_total = 0