refactor(agents): migrate .agent/ planning tree to .agents/

Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
2026-09-05 10:57:07 -04:00
parent 766702c750
commit dbf2af26c6
1118 changed files with 664 additions and 664 deletions
@@ -0,0 +1,149 @@
# Phase 69 — Full Source Removal: Files, Index, and a Confirmation Modal
**Source:** owner request (chat, 2026-09-02) — "Removing sources doesn't remove the
data from the filesystem or the rag… I need files cleaned up and the rag index
automatically synced. When I remove a source it should be totally removed. For that
reason, there should be a confirmation modal that pops up asking for confirmation if
the user clicks delete on a source."
**Story:** n/a (owner request from chat — full source removal, 2026-09-02)
**Context:**
- `app/api/git_sources.py` — `delete_git_source` is **row-only** today: its docstring
says "Removing does not touch the clones or the index — the next Sync
(`prune=True`) prunes the dropped repo (phase scope boundary)". The module's
"Scope boundary" paragraph repeats it. 404 (unknown id) / 422 (bad uuid) / 204
pins.
- `app/rag/importer.py` — `_prune` deletes a source's `Document` rows whose files are
no longer walked; `Document.chunks` carries `cascade="all, delete-orphan"`
(`app/models.py:91`), so deleting a document drops every chunk row including its
pgvector embedding. Sync source naming: git rows index under
`repo_name(row.url)` (`scripts/import_docs.py`), local rows under
`Path(row.path or row.url).expanduser().name` — the exact expressions removal must
reuse to find the right documents.
- `app/api/sync.py::_run_sync` — the pipeline whose pieces removal reuses (not
re-implements): the `bump_sources_version` short-lived-session pattern
(`app/rag/sources_meta.py`, phase 53 saved-chat invalidation — any KB change,
including a pure prune, bumps exactly once) and the change-gated
`regenerate_overview` (`app/rag/overview.py`, phase 31 — best-effort by design: an
`LLMError` returns `False` with the previous row intact).
- On-disk layout: git checkouts live under `settings.sources_dir`
(`~/bor-sources/<repo>/`), unpacked uploads under `settings.upload_dir`
(`~/bor-sources/uploads/<name>/`) — both **app-managed**. A `kind='local'` row may
also point at any owner directory; those are **not** app-managed and must never be
deleted (the page no longer offers the local-dir form — phase 49 — but the API
still accepts it, and such rows can exist).
- `frontend/assets/git-sources.js` — the remove flow calls `window.confirm` (the
**only** `window.confirm` in the frontend); module docstring bullets "remove"
(L69–76) and "Scope boundary" (L82–87) carry the stale "prunes on the next sync"
copy. `frontend/git-sources.html` — `#git-sources-hint` (L237–243) carries the same
stale copy.
- `tests/e2e/test_git_sources_admin.py` — test 4 pins the remove lifecycle through a
Playwright `page.on("dialog")` handler (accept → one DELETE; dismiss → none).
- `tests/integration/test_git_sources_api.py` — the existing DELETE pins
(`test_delete_removes_row_and_falls_back_to_env`, `test_delete_unknown_id_returns_404`,
`test_delete_invalid_id_returns_422`). `tests/integration/test_git_sources_upload.py`
is the house pattern for pointing the router at tmp dirs (the `_point_at`
monkeypatch) and faking the LLM (`FakeEmbedder`, spied `import_sources`).
- `README.md` — git-sources section (~L129–142: "Adding/removing does not clone…")
and local-sources section (~L404–410: removal semantics) carry the stale contract.
## Objective
Removing a source (admin page or API) is a **total removal**: the stored row, every
indexed document of that source (chunks + embeddings), and — for app-managed sources
— the files on disk (the git checkout or the unpacked upload folder), all in one
action. The page confirms the removal first through an accessible, in-app modal
(replacing `window.confirm`) that spells out exactly what will be deleted.
## Dependencies
- `68_search_tool` (complete; ordering by number — no functional dependency).
- Functional foundations, all complete: `28_git_based_sources` / phase 35 (sources
registry + CRUD), `38_local_directory_sources` (local rows), phase 49/`64_sync_upload_progress`
(archive uploads + upload dir), phase 32 (sync + `prune=True`), `53_stale_saved_chats`
(sources version), phase 31 (KB overview).
## Tasks
1. `01_full_removal_backend.md` — the `app/rag/source_removal.py` helper (source-name
resolver, managed-dir mapping, sibling guard, disk removal) and the rewired
`DELETE /api/git-sources/{id}` (row + index + managed files + version bump +
best-effort overview).
2. `02_confirmation_modal.md` — the accessible confirmation modal on
`/git-sources.html` (replaces `window.confirm`), the updated hint-box + docstring
copy, frontend unit pins.
3. `03_e2e_and_commit.md` — the dedicated E2E suite (modal → API → disk + DB),
`test_git_sources_admin.py` updated to the modal, README copy, full gates, commit.
## Testing & Quality
- Unit: `tests/unit/test_source_removal.py` — the resolver (git URL shapes incl.
`.git` suffix + scp-style `git@`, `~` expansion), the managed-dir mapping (git /
upload-under-root / foreign-local → `None`; the containment check so a sibling
named `uploads-foo` never counts as under `upload_dir`), disk removal (absent dir
no-op, present dir removed, `OSError` logged not fatal), the sibling-guard
decision.
- Unit (frontend): `tests/unit/test_remove_confirm_modal.py` — `window.confirm` gone
from `git-sources.js`; the dialog ids + `role="alertdialog"` + aria wiring +
Esc/Cancel/focus-return wiring present; the stale "stays indexed until the next
sync" copy gone from `git-sources.html`/`.js`; the new hint copy present.
- Integration: `tests/integration/test_source_removal_api.py` — the full-removal
matrix (task 01 step 3), tmp dirs + faked/spied LLM per the
`test_git_sources_upload.py` patterns; existing 404/422/204 pins stay green.
- E2E (mandatory, house rule): `tests/e2e/test_source_removal_cleanup.py`, run in
isolation — the modal flow end-to-end for uploaded, git, and local-directory
sources, incl. disk assertions (same-host `pathlib` against the settings-resolved
dirs) and the cancel/Esc paths.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `DELETE /api/git-sources/{id}` removes the row **and** prunes the source's
documents (+chunks/embeddings) in one commit, deletes the app-managed on-disk
dir when present (git checkout / upload folder), bumps `sources_version` when
docs were pruned, and best-effort-regenerates the overview when pruned > 0.
404/422/204 pins unchanged. Foreign local directories are never touched.
Sibling rows sharing a source name keep their documents and files.
- [ ] `/git-sources.html` removal is a page-local `role="alertdialog"` modal (no
`window.confirm` anywhere in `frontend/`): names the source, states the
removal policy, Cancel/Esc/backdrop cancel, "Removing…" lifecycle, in-modal
`role="alert"` error, focus return to the trigger, WCAG 2.1 AA basics
(labelled, focus-visible, contrast ≥4.5:1, ≥44px targets).
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
`uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_source_removal_cleanup.py -v --no-cov` green in
isolation (DB up); regression suites green in isolation:
`test_git_sources_admin.py`, `test_archive_upload_sources.py`.
- [ ] README removal-semantics copy updated (git-sources + local-sources sections).
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
`.agents/phases/complete/`.
## Locked decisions
- **Owner (chat, 2026-09-02):** removal is a total removal — row + RAG index +
app-managed files, **immediately** (not deferred to the next sync); and a real
confirmation modal (not `window.confirm`) pops up when Remove is clicked, stating
what will be deleted before it happens.
- **DB first, disk second.** The row + document prune commit atomically first (the
RAG is always consistent with the registry — this is the owner's core ask); the
disk removal runs **after** the commit, and an `OSError` is logged
(`logger.exception`) but does not fail the 204. A leftover dir is inert (no row →
never imported) and self-heals on re-add (git re-clones —
`clone_or_pull` clones when `.git` is absent; a re-upload recreates the folder).
The reverse order is forbidden: a disk failure must never leave a row pointing at
deleted files.
- **App-managed files only.** Removal deletes `sources_dir/<repo>/` for git rows and
the stored dir for local rows **when it is `upload_dir` itself or nested under it**
(containment via resolved paths + `parents`, so `…/uploads-foo` never counts). Any
other `kind='local'` path — the owner's own directory — is never touched on disk;
only its row + index entries are removed.
- **Sibling guard.** If another stored row resolves to the same source name (e.g.
`https://e.com/r` and `https://e.com/r.git` → both `r`), only the row is deleted —
the shared documents and files still belong to the sibling. Logged loudly.
- **Version bump + overview gates.** `sources_version` bumps exactly once when
pruned > 0 (the phase-53 saved-chat invalidation gate, same as sync). Overview
regeneration runs when pruned > 0 — deliberately broader than sync's
added+updated gate, because a whole-source removal changes the KB's face — and is
best-effort: an LLM failure logs and never fails the delete (the next
added/updated change refreshes it, as today).
- **Contract preserved.** `DELETE` keeps 204 with no body (the UI success path and
the 404/422 pins are unchanged); the per-operation INFO log line (PLAN §9 /
AGENTS.md rule 10) carries the counts.
## Commit
```bash
git add -A .agents/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(sources): removing a source deletes its files and index entries behind a confirmation modal"
```
@@ -0,0 +1,115 @@
# Task 01 — Full-Removal Backend: Row + Index + App-Managed Files
**Phase:** `69_source_removal_cleanup` · **Story:** n/a (owner request from chat, 2026-09-02)
## Objective
`DELETE /api/git-sources/{id}` becomes a total removal: the stored row, the source's
indexed documents (chunks + embeddings), the app-managed on-disk directory, the
sources-version bump, and a best-effort overview refresh — via a new unit-testable
helper module, keeping the 204/404/422 contract.
## Work
1. `app/rag/source_removal.py` (new module — stdlib + `app`/`scripts` imports only,
**no FastAPI**, so it unit-tests with plain objects and `tmp_path`):
- `resolve_source_name(row: GitSource) -> str` — exactly how sync/importer label
documents: `kind='git'` → `repo_name(row.url)` (import from
`scripts.import_docs`, the phase-28 helper); `kind='local'` →
`Path(row.path or row.url).expanduser().name` (the same expression
`app/api/sync.py::_run_sync` walks).
- `managed_dir_for(row: GitSource, sources_dir: Path, upload_dir: Path) -> Path | None`
— git → `sources_dir.expanduser() / repo_name(row.url)`; local → the expanded
stored path **only when** it equals `upload_dir.expanduser()` or is nested under
it (containment via `.resolve()` + `Path.parents` — a sibling named
`uploads-foo` must never count); any other local path → `None` (owner's own
directory, never touched).
- `remove_managed_dir(directory: Path | None) -> bool` — `None` or absent →
`False` (no-op, no filesystem write); present → `shutil.rmtree(directory)`;
`OSError` → `logger.exception` + `False`. **Never raises.** Returns `True`
only when a dir was actually removed.
- `has_sibling(db: Session, row: GitSource) -> bool` — another `git_sources`
row (different id) whose `resolve_source_name` equals the row's (the table is
tiny — resolve in Python, no SQL trickery).
2. `app/api/git_sources.py::delete_git_source` — rewire, in this locked order:
1. `db.get(GitSource, source_id)` → 404 `git source not found` (unchanged).
2. `name = resolve_source_name(row)`; if `has_sibling(db, row)` → `logger.warning`
(names the row + the shared source name), delete **only the row**, commit,
204 (sibling guard — the shared documents + files stay).
3. **DB first:** in the request transaction, `db.delete(doc)` for every
`Document` with `source == name` (the `all, delete-orphan` cascade drops all
chunks incl. embeddings — `app/models.py:91`), count them (`docs_pruned`),
`db.delete(row)`, `db.commit()`. A DB failure propagates as 500 **before any
disk work** (the 204 contract below never sees a half-removal).
4. **Disk second:** `files_removed = remove_managed_dir(managed_dir_for(row,
Path(get_settings().sources_dir), Path(get_settings().upload_dir)))`.
5. When `docs_pruned > 0`: make the endpoint `async def` (the
`update_document_summary` precedent in `app/api/docs.py` — async endpoint,
sync `get_db` session, `LLMClient`) and wrap
`await regenerate_overview(LLMClient())` in `try/except Exception` (log; an
LLM outage never fails the 204; `regenerate_overview` is already
`LLMError`-safe), then the phase-53 pattern: a short-lived `SessionLocal()`
(open → `bump_sources_version(db)` → `db.commit()` → `db.close()` in
`finally`, exactly as `_run_sync` does) — order: overview **then** bump,
mirroring sync (the bump lands even if the best-effort overview fails).
6. One INFO log line (PLAN §9 / AGENTS.md rule 10):
`source removed: kind=%s name=%s docs_pruned=%d files_removed=%s
overview=%s total_ms=%d` (`files_removed` is `yes|no|skipped` — `skipped`
for the sibling guard and for foreign local dirs).
- Keep the route contract/status: 204 `Response`, `Depends(require_admin)`,
uuid 422 pin (the `def` → `async def` change is the only signature delta). Update the endpoint docstring (total-removal semantics + the
locked order) and the module docstring's "Scope boundary" paragraph (removal
now performs the cleanup; Sync's `prune=True` stays for **upstream file
churn**, not for row removal).
3. `tests/integration/test_source_removal_api.py` (new) — house patterns from
`tests/integration/test_git_sources_upload.py`: `_point_at`-style monkeypatch of
the settings the router reads so `sources_dir`/`upload_dir` point at fresh
`tmp_path` dirs per test; `FakeEmbedder` / spied `regenerate_overview` where the
LLM would be hit; the real-DB `db` fixture (skip when Postgres is down — shared
conftest behavior). Seed rows via `SessionLocal` (the `test_git_sources_admin.py`
pattern) and seed `Document`/`Chunk` rows for prune assertions. Cases:
- git row + tmp checkout dir with a marker file + seeded doc → DELETE 204; row
gone from `GET /api/git-sources`; **dir gone from disk** (marker included);
source absent from `GET /api/docs` (admin); `sources_version` bumped
(`app.rag.sources_meta.current_sources_version`).
- git row, checkout dir **absent** → 204, no error (the no-op path).
- local row under the tmp `upload_dir` + seeded doc → 204; row + doc gone;
**upload folder gone from disk**.
- local row at a `tmp_path` dir **outside** `upload_dir` with a marker file +
seeded doc → 204; row + doc gone; **dir + marker file still present**.
- containment edge: `upload_dir = tmp/"u"`, local row at `tmp/"u-evil"` →
`managed_dir_for` is `None`, dir untouched (also pinned in the unit test).
- sibling guard: URLs `https://example.com/reese/r` and
`https://example.com/reese/r.git` (same `repo_name`) + docs under source `r` +
the shared checkout dir → delete the first: 204, row gone, **docs + dir
remain**; delete the second: 204, docs + dir **gone**.
- overview spy: pruned > 0 → called exactly once; a row with **no** docs → not
called; spy raising `LLMError` → still 204 (best-effort).
- version bump: docs pruned → bumped exactly once (the `sources_meta` counter);
no docs → not bumped.
4. `tests/integration/test_git_sources_api.py` — harden
`test_delete_removes_row_and_falls_back_to_env`: monkeypatch the router's
`sources_dir`/`upload_dir` at `tmp_path` (the row's example.com checkout dir
never exists, but the test must not depend on the operator's real
`~/bor-sources` being clean — zero risk of an rmtree aimed at a real checkout).
The 404/422 pins are untouched.
## Testing & Quality
- Unit (new `tests/unit/test_source_removal.py`): `resolve_source_name` (https
`.git`, https bare, scp-style `git@host:repo.git`, local with `~` and with
trailing-slash paths); `managed_dir_for` (git mapping; upload dir itself; nested
upload; foreign local → `None`; the `u-evil` containment edge);
`remove_managed_dir` (None → False; absent → False; present tree → False-free
removal + True; a read-only-ish failure path if cheaply simulatable — otherwise
the `logger.exception` branch via a monkeypatched `shutil.rmtree` raising
`OSError`); `has_sibling` (same-name sibling → True; different → False;
self-excluded).
- Coverage: **>90%** on `app/rag/source_removal.py` and the modified endpoint.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_source_removal.py tests/integration/test_source_removal_api.py -v --no-cov`
green (DB up).
- [ ] `uv run pytest` green (existing 404/422/204 DELETE pins included);
`uv run pytest --cov=app` TOTAL >90%; `uv run ruff check . && uv run pyright`
clean.
- [ ] DELETE follows the locked order (row + prune committed before any disk
work; sibling guard; best-effort overview; one bump when pruned > 0).
- [ ] No behavior change in sync / upload / import (their suites green untouched).
@@ -0,0 +1,122 @@
# Task 02 — Confirmation Modal on the Git Sources Page
**Phase:** `69_source_removal_cleanup` · **Story:** n/a (owner request from chat, 2026-09-02)
## Objective
Replace the `window.confirm` remove flow on `/git-sources.html` with an accessible,
in-app confirmation modal that names the source and states the full-removal policy,
and update every stale "prunes on the next sync" copy the phase supersedes.
## Work
1. `frontend/git-sources.html` — static modal markup inside `#git-sources-content`
(hidden by default; static markup so E2E gets stable selectors — the
`#git-sources-hint` / gate markup convention):
- `#remove-confirm-dialog` — `role="alertdialog"`, `aria-modal="true"`,
`aria-labelledby="remove-confirm-title"`, `aria-describedby="remove-confirm-copy"`,
`hidden` initially, containing:
- `#remove-confirm-title` (`<h2>`): "Remove this source?"
- `#remove-confirm-source` — a `<code>` for the source's value (git URL or local
path; **always set via `textContent` — the credential-masking discipline:
URLs may embed `user:pass@`**, phase 32).
- `#remove-confirm-copy` — a `<p>` with the locked policy text (below).
- `#remove-confirm-error` — a `<p role="alert">`, `hidden` initially.
- `#remove-confirm-cancel` (`<button type="button">`, "Cancel") and
`#remove-confirm-remove` (`<button type="button">`, destructive style class,
"Remove source").
- Update `#git-sources-hint` (currently L237–243: "…removing a source prunes its
documents from the index on the next sync") to the new contract: removing a
source immediately removes its entry, its indexed documents, and — for git
clones and uploaded archives — its files from disk (a confirmation modal spells
this out); the Sync button still mirrors the remaining sources (upstream churn
still prunes on that run).
2. `frontend/assets/git-sources.js` — rework the remove flow (the per-row Remove
button now opens the modal instead of calling `window.confirm`):
- `openRemoveConfirm(s, triggerBtn)` — populate `#remove-confirm-source`
(`textContent` = the row value, `s.path ?? s.url` for local rows, `s.url` for
git — the same `value` expression `makeRow` uses), clear
`#remove-confirm-error`, unhide the dialog, and **focus
`#remove-confirm-cancel`** (the safe default for a destructive action); record
`triggerBtn` for focus return.
- While open: `Escape` (keydown on the dialog/document) and the Cancel button
and a click on the dim backdrop all close it as **cancel** — hide the dialog,
clear the error line, return focus to `triggerBtn`, and send no request.
- `#remove-confirm-remove` click — the §7.4 never-stale lifecycle, inside the
modal: clear the error line; disable **both** buttons (Escape/backdrop
cancel are no-ops while the request is out) + relabel the confirm button
"Removing…". The in-flight state covers the whole server-side cleanup
(DB prune → file removal → best-effort overview refresh) — the same
"wait for the terminal state" pattern as the Sync/Upload processing
states, so a slow LLM refresh is expected, not a stuck button. Note in the
module docstring: navigating away mid-removal is not recommended — the row
+ index commit first, so the KB stays consistent; a rare interrupted file
step leaves an inert orphan dir (no row → never imported again).
`fetch DELETE /api/git-sources/{id}` →
- ok (204): close the dialog (focus return),
`announce("Source removed — its files and index entries were cleaned up.")`
(the existing `#git-sources-announcer`), `loadSources()`.
- non-2xx: `#remove-confirm-error` = `await apiDetail(r, "Could not remove the
source — try again.")`, re-enable both buttons + relabel ("Remove source"),
dialog stays open (the fix is one retry, not a re-search for the row).
- network failure: fixed line "Could not remove the source — is the app
reachable?" + both buttons re-enabled.
- Update the module docstring: the "remove" bullet (L69–76) and the "Scope
boundary" paragraph (L82–87) — removal now performs the full cleanup
server-side (row + index + app-managed files); the modal states that; the
page's hint box matches.
3. `frontend/assets/styles.css` — modal styling using existing design tokens only
(no CDN — AGENTS.md rule 6): a fixed full-viewport dim backdrop + a centered
dialog card (max-width ~46rem chat-column width or narrower); destructive button
aligned with the existing error-token pairs (`.tuning-ne` / `.steering-ne`
convention — text contrast ≥4.5:1); the house 3px `:focus-visible` outline on
both buttons and the dialog (focus lands visibly on Cancel at open); both
buttons ≥44px tall; no new animation (reduced-motion safe by construction).
4. `tests/unit/test_remove_confirm_modal.py` (new — the source-parsing house pattern
of `test_stale_ui_copy.py` / `test_summary_edit_ui.py`):
- `window.confirm` is **absent** from `frontend/assets/git-sources.js` (and
nowhere else in `frontend/`).
- `git-sources.html` carries `#remove-confirm-dialog` with
`role="alertdialog"`, `aria-modal`, `aria-labelledby`, `aria-describedby`, and
the `#remove-confirm-title` / `#remove-confirm-source` / `#remove-confirm-copy`
/ `#remove-confirm-error` (`role="alert"`) / `#remove-confirm-cancel` /
`#remove-confirm-remove` ids.
- `git-sources.js` wires the lifecycle: focus-on-open (Cancel), Escape handling,
backdrop cancel, the "Removing…" in-flight label, the success announce string,
focus return to the trigger, and `textContent` population of
`#remove-confirm-source` (no `innerHTML` on that node).
- Stale-copy pins: "stays indexed until the next sync" and "prunes … on the next
sync" absent from `git-sources.js` + `git-sources.html`; the new hint copy
present in `git-sources.html` (README is pinned in task 03).
## Locked decisions
- **Modal copy** (the `#remove-confirm-copy` text, verbatim):
"This permanently removes the source entry, all of its indexed documents from the
knowledge base, and — for git clones and uploaded archives — the files on the
server's disk. Files in your own local directories are never touched. This cannot
be undone." One fixed paragraph for both kinds (the UI does not know whether a
local row is an upload or the owner's own directory — the fixed policy text is
accurate for both; naming the source above it makes the target unambiguous).
- Cancel is the safe default: focus lands on **Cancel** at open; Escape and
backdrop clicks cancel; only the explicit "Remove source" button sends the
request. While the DELETE is in flight both buttons disable (no second
request, no half-cancel of an in-progress server-side removal). Focus returns
to the row's Remove button on any close (WCAG 2.1).
- The modal is page-local (no shared component extracted — one caller; extraction
is a later-phase concern if a second destructive flow appears).
## Testing & Quality
- Unit: `tests/unit/test_remove_confirm_modal.py` (pins above); the a11y
interaction assertions (focus, Escape, ≥44px, focus-visible) are E2E-pinned in
task 03.
- Coverage: no `app/` code changes in this task — the `>90%` gate stays green as
part of the full suite run.
## Completion Criteria
- [ ] `rg window.confirm frontend/` → no matches.
- [ ] Open (focus on Cancel) → confirm → "Removing…" → success path closes the
modal, announces, and reloads the list; failure path shows the in-modal
`role="alert"` line and re-enables the button; Cancel/Esc/backdrop close
without a request and return focus to the trigger.
- [ ] `uv run pytest tests/unit/test_remove_confirm_modal.py -v --no-cov` green;
`uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] `#git-sources-hint` + module docstring carry the new contract; no stale
"next sync" removal copy in `git-sources.html` / `git-sources.js`.
@@ -0,0 +1,112 @@
# Task 03 — E2E Suite, Regression Update, README Copy, Commit
**Phase:** `69_source_removal_cleanup` · **Story:** n/a (owner request from chat, 2026-09-02)
## Objective
Prove the full removal end-to-end (modal → API → disk + DB) in a dedicated
isolated Playwright suite, update the existing admin E2E to the modal, fix the
README's removal copy, and pass the full phase gates with one commit.
## Work
1. `tests/e2e/test_source_removal_cleanup.py` (new — house docstring header:
"Phase 69 story E2E … run in isolation (DB must be up): `uv run pytest
tests/e2e/test_source_removal_cleanup.py -v --no-cov`", test → mapping list):
- Setup patterns from the existing suites: session app + mock LLM via
`tests/e2e/conftest.py`; `login(page, app_url, next=GIT_SOURCES_URL)`
(the `test_archive_upload_sources.py` form-login helper); in-test archive
built with `tarfile` (that suite's L190 pattern); rows + `Document` rows
seeded through `SessionLocal` (`test_git_sources_admin.py` pattern);
**disk paths resolved exactly like the app**: `Path(get_settings().sources_dir).expanduser()`
/ `…(upload_dir)` — the E2E conftest does not override those vars and pytest
runs with `cwd=REPO`, so the test process and the app process resolve the
same `.env` (the test process also sets the same forced env the conftest
sets for the app where it matters — sources/upload dirs are NOT among them).
- The suite triggers **no sync** (git rows are `example.com` URLs, never
cloned; the only real artifact is the API-driven upload of a small archive
with a unique name `phase69-<8-hex-chars>.tar.gz` containing one `.md` file).
- Mapped tests:
1. `test_uploaded_source_removal_cleans_index_and_disk` — upload the
uniquely named archive via the logged-in `page.request` POST
(`/api/git-sources/upload`, 202), poll `GET /api/git-sources/upload/status`
to `success`; assert the folder
`<upload_dir>/<name>/` exists on disk (pathlib) and the source's document
appears in `GET /api/docs` (admin); then in the UI: that row's Remove →
modal visible (`role="alertdialog"`, `#remove-confirm-source` text = the
upload path, focus on `#remove-confirm-cancel`) → click "Remove source" →
"Removing…" state → row gone; `GET /api/docs` shows no document for the
source; **the folder is gone from disk**; the announcer carries the
success line.
2. `test_git_source_removal_removes_checkout_and_index` — seed a git row
(`SessionLocal`, deterministic `https://example.com/reese/phase69-gone.git`
URL), create the checkout dir
`<sources_dir>/<repo_name(url)>/` with a marker file, seed a `Document`
row for source `repo_name(url)`; UI delete via the modal → row gone,
**checkout dir gone from disk** (marker included), document gone from
`GET /api/docs`.
3. `test_git_source_removal_without_checkout_succeeds` — seed the row, no
checkout dir (never synced) → UI delete → row gone (the absent-dir
no-op path; no error state anywhere on the page).
4. `test_local_directory_source_files_never_deleted` — create a `tmp_path`
user dir with a marker file, seed a `kind='local'` row
(`path=str(dir)`) + a `Document` row for the dir's basename source; UI
delete → row gone, document pruned from `GET /api/docs`, **dir + marker
file still present** (pathlib).
5. `test_remove_modal_cancel_and_esc_keep_everything` — (a) Remove → modal →
click Cancel → row stays, **zero DELETE requests** (`page.on("request")`
tracker, the `test_git_sources_admin.py` pattern), document remains;
(b) re-open → `page.keyboard.press("Escape")` → dialog hidden, focus back
on the trigger button, still zero DELETEs.
6. `test_remove_modal_a11y_and_no_cdn` — dialog aria attributes
(`aria-modal`, `aria-labelledby`, `aria-describedby`); `:focus-visible`
3px outline computable on both buttons; both buttons ≥44px tall; after a
successful removal the `#git-sources-announcer` (role=status) carries the
success line; the page loads only same-origin resources (AGENTS.md
rule 6 — the no-CDN pin pattern from `test_git_sources_admin.py`).
2. `tests/e2e/test_git_sources_admin.py` — update test 4's remove section and the
module docstring: replace the `page.on("dialog")` handler with modal
interactions — accept = click `#remove-confirm-remove`; cancel = click
`#remove-confirm-cancel`. Keep the request-tracker assertions (exactly one
DELETE on accept; none on cancel) and the "seed row survived" checks. Remove
the `Dialog` import if now unused. Everything else in the suite is untouched.
3. `README.md` — update the stale removal contract (the unit copy-pins cover the
frontend; the README is checked here):
- git-sources section (~L129–142: "Adding/removing does not clone …" and the
"a removed source's documents leave the index" line) and local-sources
section (~L404–410: "Removing the row on the page stops the directory being
a source; its … [documents leave the index on the next sync]") → the new
contract: removing a source (a confirmation modal states it first)
immediately removes its entry, its indexed documents, and — for git clones
and uploaded archives — its files from disk; files in the owner's own local
directories are never touched; Sync still prunes upstream file churn.
4. Gates + commit (in order):
- `uv run pytest --cov=app --cov-report=term-missing` — green, TOTAL **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
- `uv run pytest tests/e2e/test_source_removal_cleanup.py -v --no-cov` — green
in isolation (DB up: `podman compose up -d db`).
- Regression suites, each in isolation:
`uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`,
`uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov`.
- Move the phase dir: `mv .agents/phases/todo/69_source_removal_cleanup
.agents/phases/complete/`.
- Commit (AGENTS.md rule 8):
`git add -A .agents/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(sources): removing a source deletes its files and index entries behind a confirmation modal"`
## Testing & Quality
- The dedicated E2E file (6 mapped tests) **is** this phase's Playwright suite —
it runs in isolation per AGENTS.md rules 4/9 and covers the backend cleanup
(disk + DB), the modal UX, the cancel/Esc paths, and the a11y/no-CDN basics.
- Coverage: **>90%** on `app/` (the full-suite gate above).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_source_removal_cleanup.py -v --no-cov` green in
isolation; all six mapped behaviors hold, incl. the three disk assertions
(upload folder removed, checkout removed, foreign local dir untouched).
- [ ] `test_git_sources_admin.py` green in isolation with the modal (no Playwright
`dialog` handler left in it); `test_archive_upload_sources.py` green in
isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL >90%;
`uv run ruff check . && uv run pyright` clean.
- [ ] README removal copy matches the implemented contract (no "next sync"
removal semantics left in the two sections above).
- [ ] One `--no-gpg-sign` commit with the phase dir in
`.agents/phases/complete/69_source_removal_cleanup/`.