chore(agent): phase roadmap from TODO.md — 4 phases (lite document summaries, KB overview in prompt, admin sync button, cache busting)
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
# Phase 33 — Cache Busting (un-stick the pages)
|
||||
|
||||
**Source:** `TODO.md L6 — "We need better cache busting, the pages are too sticky"`
|
||||
**Story:** `.agent/user_stories/cache-busting.md`
|
||||
**Context:** `app/main.py` serves the whole `frontend/` directory through one `StaticFiles(html=True)` catch-all mount; the five HTML pages reference assets **without any version** (`href="/assets/styles.css"`, `src="assets/markdown.js"`, `src="/assets/app.js"`, …), so browsers happily keep stale CSS/JS/HTML after a deploy — the "too sticky" report. The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) asserts no `https://` references — appending `?v=` keeps every reference same-origin, so it stays green. SSE/API live under `/api/*` and must be untouched (SSE already ships `Cache-Control: no-cache` itself).
|
||||
|
||||
## Objective
|
||||
A deploy must be visible without a hard refresh: HTML pages are **always revalidated** (`Cache-Control: no-cache`) and reference their assets with a version token (`?v=<token>`); assets are served **immutable for 1 year** (the token in the URL identifies the content, so long caching is safe). Zero new services, zero build-step changes, no CDN.
|
||||
|
||||
## Dependencies
|
||||
- `32_admin_sync_button` (todo) — sequencing only; no shared code (this phase is transport-layer and independent of the RAG work).
|
||||
|
||||
## Tasks
|
||||
1. `01_asset_version_token.md` — `app/core/caching.py::asset_version()`: git short SHA (homelab checkouts have a `.git`), stable mtime+size content-hash fallback, computed once per process.
|
||||
2. `02_caching_middleware.md` — the response middleware (HTML `no-cache` + `?v=` rewrite; `/assets/*` immutable) wired into `create_app` + integration tests.
|
||||
3. `03_e2e_and_docs.md` — `tests/e2e/test_cache_busting.py` (Playwright header assertions), README, story file, commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: version token (git path, fallback path, failure path), the asset-reference rewrite (both `href`/`src` and leading-slash-less `assets/…` refs, no double-`?v=`).
|
||||
- Integration: page headers + rewritten references; asset headers; no-CDN test green; SSE endpoint responses untouched (existing chat SSE tests green).
|
||||
- Coverage: **>90%** on `app/` (the new `app/core/caching.py` fully covered); TOTAL ≥ pre-change.
|
||||
- E2E (mandatory, A16): `tests/e2e/test_cache_busting.py` — one story, run **in isolation**; real Chromium asserting the headers and the versioned request URLs the browser actually makes.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Every HTML page (`/`, `/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) is served with `Cache-Control: no-cache` and its asset references carry `?v=<token>` (token non-empty, stable across requests, changes when the frontend content changes).
|
||||
- [ ] `/assets/*` responses carry `Cache-Control: public, max-age=31536000, immutable`.
|
||||
- [ ] `/api/*` (incl. the SSE chat stream) responses are byte-for-byte header-wise unaffected beyond what they already send; no-CDN integration test green.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
|
||||
- [ ] `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green in isolation; `test_smoke.py` + one RAG E2E stay green.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `.agent/user_stories/cache-busting.md` exists; README documents the caching behavior + how the token changes on deploy.
|
||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching`); `.agent/phases/todo/33_cache_busting/` moved to `.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Version token** — `asset_version()`: if the project checkout has a `.git` (the homelab reality), the token is `git rev-parse --short HEAD` (a commit = a deploy, so the token flips on every deploy); otherwise a stable hash of the frontend tree (sorted `relpath + mtime_ns + size`, first 12 hex chars) so dev checkouts still bust. Computed **once per process** (`lru_cache`) — zero per-request git/file cost.
|
||||
- **Rewrite scope** — only the five known HTML pages are rewritten (a small regex over `href="…assets/…"` / `src="…assets/…"` appending `?v=` when absent). No templating layer, no build step, no changes to the static files themselves (the `Containerfile` esbuild stage is untouched).
|
||||
- **Asset caching** — `/assets/*` are cached `immutable` for 1 year **because** the URL carries the token; the unversioned path keeps working (StaticFiles ignores the query string), so old tabs and tests referencing `/assets/x.js` directly still resolve.
|
||||
- **Middleware boundary** — the middleware touches exactly two shapes: the five page paths (body rewrite + `no-cache`) and `/assets/*` (header only). Everything else — all `/api/*` including SSE — passes through byte-identical (SSE keeps its own `no-cache`). Implemented as a Starlette middleware that only rewrites `text/html` responses under the page paths; if `Response.body()` turns out to misbehave on the `FileResponse` streaming path, the fallback is five explicit FastAPI routes that read + rewrite the files (identical observable behavior — the executor picks whichever passes the tests).
|
||||
- **A11 untouched** — no CDN, no new packages, no new services (A12 untouched).
|
||||
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 01 — app/core/caching.py: the asset version token
|
||||
|
||||
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "We need better cache busting, the pages are too sticky"`
|
||||
**Story:** `.agent/user_stories/cache-busting.md`
|
||||
|
||||
## Objective
|
||||
One source of truth for the asset version token: git short SHA when the checkout is a repo (a commit = a deploy), a stable content-hash fallback otherwise — computed once per process.
|
||||
|
||||
## Work
|
||||
1. `app/core/caching.py` (new) —
|
||||
- `def asset_version(static_dir: str | None = None) -> str`:
|
||||
- `static_dir` defaults to `get_settings().static_dir` (resolves to `frontend`); the git repo root is its parent.
|
||||
- **Git path:** if `(static_dir parent / ".git")` exists → `subprocess.run(["git", "-C", str(root), "rev-parse", "--short", "HEAD"], capture_output=True, text=True, timeout=5)` → the short SHA (e.g. `3841bd5`).
|
||||
- **Fallback / failure path** (no `.git`, git missing, non-zero exit, timeout, unreadable): `hashlib.sha256` over the sorted list of `f"{relpath}:{mtime_ns}:{size}"` for every regular file under `static_dir`, first **12 hex chars**. A missing/empty `static_dir` → `"dev"`.
|
||||
- `@functools.lru_cache(maxsize=None)` on the resolved-argument wrapper (settings are process-stable; the token must not be recomputed per request). Document that a process restart or new commit changes the token.
|
||||
2. `tests/unit/test_caching.py` (new):
|
||||
- git path: a `tmp_path` repo (`git init -q` + a commit of a dummy file, with a `frontend/` subdir inside) → token == `git rev-parse --short HEAD` of that repo; second call returns the same value (cache).
|
||||
- fallback: a plain `tmp_path/frontend` with two files → 12-hex token; unchanged tree → same token; touch/modify a file (mtime or size change) + `asset_version.cache_clear()` → different token.
|
||||
- failure: `static_dir` with a `.git` present but `git` removed from PATH (monkeypatch `subprocess.run` to raise `FileNotFoundError`) → falls back to the content hash, no exception.
|
||||
- empty/missing dir → `"dev"`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: Work step 2.
|
||||
- Coverage: **>90%** on `app/core/caching.py` (all branches).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] All unit tests green (git, fallback, failure, empty — with `cache_clear()` between parametrized cases).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 02 — Caching middleware + wiring + integration tests
|
||||
|
||||
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "the pages are too sticky" (HTML revalidation + versioned asset URLs + immutable assets)`
|
||||
**Story:** `.agent/user_stories/cache-busting.md`
|
||||
|
||||
## Objective
|
||||
Apply the caching behavior at the transport layer: HTML pages `no-cache` with `?v=<token>` on every local asset reference; `/assets/*` immutable 1-year; everything else (all `/api/*`, including SSE) untouched.
|
||||
|
||||
## Work
|
||||
1. `app/core/caching.py` (extend task 01's module):
|
||||
- `HTML_PAGES: tuple[str, ...] = ("/", "/index.html", "/sources.html", "/document.html", "/login.html", "/tuning.html")`.
|
||||
- `_ASSET_REF_RE = re.compile(r'((?:href|src)="(?:/)?assets/[^"?#]+)(")')` — matches `<link rel="stylesheet" href="/assets/styles.css">`, `<script src="assets/markdown.js"></script>` (no leading slash!), and `<script type="module" src="/assets/app.js">`; the rewrite appends `?v=<token>` before the closing quote, only when the reference has no query/hash yet (idempotent — never a double `?v=`).
|
||||
- `def rewrite_asset_refs(html: str, token: str) -> str` — the pure, unit-testable rewrite.
|
||||
- `def configure_caching(app: FastAPI) -> None` — one `@app.middleware("http")` (or equivalent Starlette middleware) that, **after** the response is produced:
|
||||
- `path.startswith("/assets/")` → `response.headers["Cache-Control"] = "public, max-age=31536000, immutable"` (header only — never touch the body).
|
||||
- `request.url.path` in `HTML_PAGES` **and** response content-type is `text/html` → `Cache-Control: no-cache` + `response.body = rewrite_asset_refs(body, asset_version())` (body via `await response.body()` — works for the buffered StaticFiles/FileResponse HTML responses; on any error or non-`text/html`, fall through to the unmodified response with only `no-cache`).
|
||||
- everything else → completely untouched (no header, no body work). The `/api/*` SSE stream in particular must not be read or rewritten.
|
||||
2. `app/main.py` — `from app.core.caching import configure_caching`; call `configure_caching(app)` inside `create_app()` after the routers (order: middleware wraps the whole app — call it before `return app`).
|
||||
3. `tests/unit/test_caching.py` (extend) — `rewrite_asset_refs`:
|
||||
- versioned: `href="/assets/styles.css"` → `href="/assets/styles.css?v=abc123"`; `src="assets/markdown.js"` (no slash) → versioned; `src="/assets/app.js"` (module script) → versioned.
|
||||
- idempotent: an already-`?v=`-tagged reference is not double-tagged; a `#fragment` or existing `?query` reference is left alone.
|
||||
- non-asset references untouched (`href="/sources.html"`, `href="data:…"`, `href="/login.html?next=…"`).
|
||||
4. `tests/integration/test_api.py` (extend):
|
||||
- `GET /` → 200, `cache-control: no-cache`; body contains `href="/assets/styles.css?v=<token>"` with a non-empty token matching `asset_version()`; the unversioned string `href="/assets/styles.css">` is **gone** from the body.
|
||||
- each of the other four pages (`/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) → `no-cache` + at least one versioned asset reference.
|
||||
- `GET /assets/styles.css` → 200, `cache-control` contains `immutable` and `max-age=31536000`.
|
||||
- `GET /api/health` → response has **no** `cache-control` injected (baseline: FastAPI's default) — assert equality with the pre-middleware behavior; the SSE chat endpoint (`tests/integration/test_chat_api.py`) stays green unmodified.
|
||||
- the existing no-CDN test (`test_html_pages_served_locally_no_cdn`) stays green — the rewritten references are still same-origin.
|
||||
5. `tests/unit/test_main.py` — app creation still succeeds with the middleware wired (existing creation tests stay green; add an assertion that `create_app()`'s middleware stack includes the caching middleware by name).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: Work steps 3 + 5; Integration: Work step 4.
|
||||
- Coverage: **>90%** on `app/core/caching.py`; `app/` TOTAL ≥ pre-change.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] All new unit + integration tests green; the full suite green (especially `test_chat_api.py` — SSE unaffected).
|
||||
- [ ] Manual check: `curl -si localhost:8000/ | grep -i cache-control` → `no-cache`; the HTML body shows `?v=…` asset refs; `curl -si localhost:8000/assets/styles.css | grep -i cache-control` → immutable.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 03 — Story E2E (Playwright header assertions) + README + commit
|
||||
|
||||
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — (whole item: better cache busting — the pages are too sticky)"`
|
||||
**Story:** `.agent/user_stories/cache-busting.md`
|
||||
|
||||
## Objective
|
||||
The story gate: a real-browser E2E that asserts what the browser actually receives (HTML `no-cache`, versioned asset request URLs, immutable asset headers), plus README docs, story file, and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_cache_busting.py` (new) — collect responses with `page.on("response")`:
|
||||
- `test_html_pages_are_no_cache_and_versioned` — navigate to `/`:
|
||||
- the document response's `cache-control` header is `no-cache`;
|
||||
- the `styles.css` request URL contains `?v=` and the response's `cache-control` contains `immutable` + `max-age=31536000`;
|
||||
- the `app.js` request URL contains the **same** token value as the CSS one (single token per process);
|
||||
- the served HTML (`page.content()`) contains no unversioned `/assets/styles.css"` reference.
|
||||
- `test_other_pages_share_the_token` — navigate to `/sources.html` then `/login.html`: each document response is `no-cache`; both pages' CSS requests carry the same token.
|
||||
- `test_api_responses_unaffected` — `page.request.get("/api/health")` → no `cache-control: no-cache`/immutable injection (the endpoint's baseline headers only); a chat SSE POST still streams (reuse the minimal chat-request helper from an existing E2E — the stream must complete with `done`).
|
||||
2. `README.md` — new short "Caching / deploys" section: HTML is always revalidated; assets are cached 1 year immutable and carry `?v=<token>`; the token is the git short SHA (falls back to a content hash in non-git checkouts) and flips on every commit/deploy — no hard refresh needed anymore; API/SSE caching is unchanged.
|
||||
3. `.agent/user_stories/cache-busting.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
|
||||
4. Commit: `git commit --no-gpg-sign -m "perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching"`; move `.agent/phases/todo/33_cache_busting/` → `.agent/phases/complete/`.
|
||||
5. **Deploy note (post-commit, owner action)**: after this phase lands, the *first* deploy also requires browsers to see the new HTML once (revalidation) — one normal navigation; thereafter every commit is picked up automatically.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` for the app boot; the mock LLM keeps the SSE check deterministic).
|
||||
- Regression in isolation: `test_smoke.py`, `test_chat_rag.py`.
|
||||
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] All E2E tests green in isolation (headers + token consistency + SSE unaffected).
|
||||
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
|
||||
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
|
||||
- [ ] `podman compose up -d` (full app) smoke: a fresh browser profile loads the site and every asset request is versioned (manual confirmation recorded in the commit message or phase notes).
|
||||
Reference in New Issue
Block a user