brain-of-reese/app/core/caching.py

"""Static-frontend cache busting (phase 33).

Two layers, one module:

* **Version token** (``asset_version()``) — the value the HTML pages
  append to their asset URLs (``?v=<token>``).
* **Response middleware** (``CachingMiddleware`` / ``configure_caching``)
  — applies the caching behavior at the transport layer: the known
  HTML pages (``HTML_PAGES``) and the dynamic share page
  ``/shared/<token>`` (phase 51) are always revalidated with a full
  200 body (``no-cache``, never a 304) publishing no validators, and
  their local asset references are rewritten to carry
  ``?v=<token>``; ``/assets/*`` is
  served ``immutable`` for a year; everything else — all of ``/api/*``,
  including the SSE chat stream — passes through byte-identical.

Why pages must never 304 (phase 54): the static file's validators
(``etag`` / ``last-modified``) describe the *file*, but the bytes the
browser receives are the *rewritten body* this process built from its
own token. A conditional request that matched those validators would
304 out of the rewrite — the browser keeps the HTML it already has,
whose ``?v=`` points at the previous commit's CSS/JS, which are cached
``immutable`` for a year. So on page paths the middleware strips
``if-none-match`` / ``if-modified-since`` from the inbound request
(before the downstream ``FileResponse`` can act on them) and drops the
outbound validators. The asymmetry is deliberate: a 304 on
``/assets/*`` is safe because the URL itself encodes the version, while
a 304 on an HTML page is never safe — the served body depends on the
process token, which the validator ignores.

Phase 91 (task 02) — the pre-paint theme tag: in the SAME rewrite
branch, AFTER the ``?v=<token>`` asset rewrite, the effective
``ui_settings`` row (task 01's :func:`app.core.theming.effective_settings`
resolver — one short-lived session per response, NO process cache: the
owner changes the theme at runtime from the admin tab, so the next
request must see it without a restart, and a single-row SELECT is
negligible at homelab page traffic) is rendered as an inline
``<style id="bor-theme">:root{…}</style>`` and inserted immediately
BEFORE the first ``</head>`` (:func:`app.core.theming.inject_theme`),
so a themed deployment paints its palette on the FIRST paint — no red
flash, no pop-in. An unset/defaults deployment gets ``tag == ""`` —
the identity no-op — and serves the EXACT pre-phase-91 rewrite-only
bytes (the byte-identical contract, B4); a DB blip (or a pre-migration
boot) is the same no-op, the page never breaks. The asset rewrite is
untouched, and ``/api/*`` / ``/assets/*`` still pass through
byte-identical.

Phase 91 (task 05, defect fix) — the CSP extension: the phase-82
policy (A1, ``default-src 'self'`` with no ``style-src``) BLOCKS the
inline tag in every real browser, so a themed HTML page's response
also carries ``style-src 'self' 'sha256-<hash>'`` appended to the A1
string, where ``<hash>`` is the CSP3 hash of the EXACT tag content
(:func:`app.core.theming.theme_csp_hash`) — the current theme is the
only inline style ever permitted (no ``'unsafe-inline'``; a different
palette or any other inline style is still blocked). The untagged
response keeps the plain A1 string (the outer
:class:`~app.core.security_headers.SecurityHeadersMiddleware`
preserves a CSP an inner layer has already set), and no non-HTML
response ever gets the extension.

The token is computed **once per process** (``functools.cache``, i.e.
``lru_cache(maxsize=None)``) — zero per-request git/file cost. It changes
when a new commit lands (git path) or the frontend tree's mtimes/sizes
change (fallback path); either way the change takes effect on the next
process start. Tests clear it with ``asset_version.cache_clear()``.

Version-token sources:

* **Git checkouts** (the homelab reality — the project root next to
  ``frontend/`` has a ``.git``): the token is ``git rev-parse --short HEAD``.
  A commit is a deploy, so the token flips on every deploy.
* **Plain directories** (dev checkouts without a ``.git``, or a git failure):
  a stable content hash — first 12 hex chars of SHA-256 over the sorted
  ``relpath:mtime_ns:size`` list of every regular file under the static dir.
  Touching or editing a file (plus a process restart) busts the cache.
* **Missing/empty directory**: ``"dev"``.
"""
from __future__ import annotations

import functools
import hashlib
import logging
import re
import subprocess
from pathlib import Path

from fastapi import FastAPI
from starlette.datastructures import MutableHeaders
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.requests import Request
from starlette.responses import Response

from app.config import get_settings
from app.core import theming
from app.core.security_headers import CSP
from app.db import SessionLocal

logger = logging.getLogger("app")

#: Token for a missing/empty static dir (e.g. a build that ships the API
#: only — there is nothing on disk to version).
DEV_TOKEN = "dev"

#: Hard cap for the ``git rev-parse`` call — a deploy must not hang a boot.
_GIT_TIMEOUT_S = 5


def _static_files(static_dir: Path) -> list[str]:
    """Sorted ``relpath:mtime_ns:size`` entries for every regular file."""
    entries: list[str] = []
    for path in static_dir.rglob("*"):
        if path.is_file():
            st = path.stat()
            rel = path.relative_to(static_dir).as_posix()
            entries.append(f"{rel}:{st.st_mtime_ns}:{st.st_size}")
    return sorted(entries)


def _git_short_sha(repo_root: Path) -> str:
    """``git -C <repo_root> rev-parse --short HEAD`` (raises on any failure)."""
    proc = subprocess.run(
        ["git", "-C", str(repo_root), "rev-parse", "--short", "HEAD"],
        capture_output=True,
        text=True,
        timeout=_GIT_TIMEOUT_S,
        check=True,
    )
    sha = proc.stdout.strip()
    if not sha:
        raise ValueError("git returned an empty short SHA")
    return sha


@functools.cache  # == lru_cache(maxsize=None): one token per process
def asset_version(static_dir: str | None = None) -> str:
    """Version token for the frontend's asset URLs (see module docstring).

    ``static_dir`` defaults to ``get_settings().static_dir`` (``frontend``);
    the git repo root is taken to be its parent. The result is cached for
    the process lifetime — clear it with ``asset_version.cache_clear()``.
    """
    target = get_settings().static_dir if static_dir is None else static_dir
    directory = Path(target).expanduser().resolve()
    files = _static_files(directory) if directory.is_dir() else []
    if not files:
        return DEV_TOKEN
    repo_root = directory.parent
    if (repo_root / ".git").exists():
        try:
            return _git_short_sha(repo_root)
        except (OSError, subprocess.SubprocessError, ValueError):
            # git missing (OSError), timeout or non-zero exit
            # (SubprocessError — e.g. a repo with no commits), or empty
            # output → the stable content-hash fallback keeps dev checkouts
            # (and broken repos) cache-busting instead of breaking a boot.
            pass
    digest = hashlib.sha256("\n".join(files).encode("utf-8")).hexdigest()
    return digest[:12]


# ---------------------------------------------------------------------------
# Response-layer cache busting (task 02)
# ---------------------------------------------------------------------------

#: The known HTML pages — the ONLY paths whose body is rewritten.
#: Phase 35 adds the admin git sources page: without this entry it
#: would serve unversioned asset refs, which the immutable-for-a-year
#: asset caching would pin to stale CSS after a deploy.
HTML_PAGES: tuple[str, ...] = (
    "/",
    "/index.html",
    "/sources.html",
    "/document.html",
    "/login.html",
    "/tuning.html",
    "/git-sources.html",  # phase 35: the admin git sources page
    "/history.html",  # phase 50: the admin saved-chats page
    "/tokens.html",  # phase 79 task 06: the admin tokens page (shell route)
    "/theme.html",  # phase 91 task 04: the admin theme page (shell route)
    # phase 51: the shared page's STATIC path (the static mount serves
    # shared.html at /shared.html as well as the real route serves the
    # dynamic /shared/<token> — both must carry the no-cache + ?v=
    # contract, so the direct URL can never pin stale assets).
    "/shared.html",
    # phase 59: the doc edit screen (the flow page task 06 ships).
    "/doc-edit.html",
)

#: Prefix of the versioned static assets (header-only caching; the body is
#: never read or modified).
ASSETS_PREFIX = "/assets/"

#: ``/assets/*`` — the URL carries ``?v=<token>``, so long-term caching is
#: safe (a deploy changes the token, hence the URL).
ASSET_CACHE_CONTROL = "public, max-age=31536000, immutable"

#: HTML pages are always revalidated — the page is the cheap redirector to
#: the (long-cached) versioned assets.
HTML_CACHE_CONTROL = "no-cache"

#: Matches ``href="…assets/…"`` / ``src="…assets/…"`` references — leading
#: slash optional (the pages mix ``/assets/…`` and ``assets/…``) — that do
#: NOT already carry a query string or a fragment: ``[^"?#]+`` stops at
#: ``?``/``#``, so an already-``?v=``-tagged reference can never match
#: again (idempotent rewrite, never a double ``?v=``).
_ASSET_REF_RE = re.compile(r'((?:href|src)="(?:/)?assets/[^"?#]+)(")')


def _is_known_page(path: str) -> bool:
    """One source of truth for the known-page contract (phase 54).

    True for ``HTML_PAGES`` and for the dynamic share page
    ``/shared/<token>`` (phase 51) by path prefix. Shared by the
    inbound conditional-header strip and the outbound rewrite branch so
    the two can never drift apart.
    """
    return path in HTML_PAGES or path.startswith("/shared/")


def rewrite_asset_refs(html: str, token: str) -> str:
    """Append ``?v=<token>`` before the closing quote of every local
    ``assets/…`` ``href``/``src`` reference.

    Pure and idempotent: references that already carry a query string or a
    fragment (``?v=…``, ``#frag``) do not match the pattern and are left
    alone, and non-asset references (``href="/sources.html"``,
    ``href="data:…"``) pass through untouched.
    """
    return _ASSET_REF_RE.sub(lambda m: f"{m.group(1)}?v={token}{m.group(2)}", html)


async def _read_body(response: Response) -> bytes:
    """Buffer the whole response body.

    starlette 1.x's ``BaseHTTPMiddleware.call_next`` returns a streaming
    response (no ``body()`` helper), so drain ``body_iterator``; a plain
    buffered ``Response`` simply carries its ``body``.
    """
    if hasattr(response, "body"):
        raw = response.body  # bytes | memoryview (Response.render's output)
        if isinstance(raw, memoryview):
            return bytes(raw)
        return raw
    # starlette 1.x's BaseHTTPMiddleware.call_next wraps the app response
    # in a streaming shell (``body_iterator``, no ``body``) that is NOT a
    # StreamingResponse subclass — drain the iterator instead.
    iterator = response.body_iterator  # pyright: ignore[reportAttributeAccessIssue]
    return b"".join([chunk async for chunk in iterator])


def _no_cache_headers(response: Response) -> dict[str, str]:
    """Copy the original headers, drop the stale ``content-length`` (the
    body size changes on rewrite), drop the upstream validators —
    ``etag`` / ``last-modified`` describe the static file, not the
    rewritten body, so a page response must never be revalidated against
    them later (phase 54) — and force ``Cache-Control: no-cache``."""
    headers = {k: v for k, v in response.headers.items()}
    headers.pop("content-length", None)
    for name in ("etag", "last-modified"):
        headers.pop(name, None)
    headers["Cache-Control"] = HTML_CACHE_CONTROL
    return headers


class CachingMiddleware(BaseHTTPMiddleware):
    """Transport-layer cache busting (phase 33, revalidation fix phase 54).

    Touches exactly two response shapes:

    * ``/assets/*`` — ``Cache-Control: public, max-age=31536000, immutable``
      (header only — the body is never read; conditional requests may
      still 304, which is safe because the URL carries ``?v=<token>``).
    * the known HTML pages (``HTML_PAGES``) plus the dynamic share page
      ``/shared/<token>`` (phase 51, by path prefix) — always revalidated
      with a full 200 body (no 304), publishing no validators: the inbound
      ``if-none-match`` / ``if-modified-since`` are stripped before the
      downstream app runs, ``Cache-Control: no-cache`` is set, the outbound
      ``etag`` / ``last-modified`` are dropped, and (for ``text/html``
      bodies) every local asset reference gains ``?v=<token>``.

    Everything else — all of ``/api/*`` (including the SSE chat stream) —
    passes through byte-identical: no header changes, the body stream is
    never drained.
    """

    async def dispatch(
        self, request: Request, call_next: RequestResponseEndpoint
    ) -> Response:
        path = request.url.path

        # Phase 54: on known page paths, strip the conditional request
        # headers BEFORE the downstream app runs. The 304 is produced
        # downstream (Starlette's ``FileResponse``/``StaticFiles`` honours
        # ``If-None-Match`` / ``If-Modified-Since`` before this middleware
        # sees a response), so a validator match would return a bodiless
        # 304 whose body can never be rewritten to carry the current
        # ``?v=<token>`` refs — the browser would keep HTML pointing at
        # the previous commit's assets, immutable-cached for a year.
        # ``MutableHeaders`` mutates ``request.scope["raw_headers"]`` in
        # place, so the downstream app only ever sees the full 200. The
        # strip is scoped to page paths only — ``/assets/*`` 304s are safe
        # (the URL is versioned) and ``/api/*`` (incl. the SSE stream)
        # must stay byte-identical.
        if _is_known_page(path):
            headers = MutableHeaders(scope=request.scope)
            for name in ("if-none-match", "if-modified-since"):
                if name in headers:
                    del headers[name]

        response = await call_next(request)

        if path.startswith(ASSETS_PREFIX):
            response.headers["Cache-Control"] = ASSET_CACHE_CONTROL
            return response

        if not _is_known_page(path):
            # /api/* (incl. SSE), /favicon.ico, unknown paths: untouched.
            return response

        # Phase 54: a bodiless downstream status (204/304) on a page
        # path is NEVER rewritten — starlette forbids a body on those
        # statuses, and the rewrite path would build exactly that.
        # Belt-and-braces: after the inbound conditional-header strip,
        # StaticFiles cannot 304 these paths; a future route or proxy
        # could still, so pass through with no-cache and no validators.
        if response.status_code in (204, 304):
            response.headers["Cache-Control"] = HTML_CACHE_CONTROL
            for name in ("etag", "last-modified"):
                if name in response.headers:
                    del response.headers[name]
            return response

        content_type = response.headers.get("content-type", "")
        if content_type.split(";", 1)[0].strip().lower() != "text/html":
            # A page path with a non-HTML body (e.g. the 404 JSON served
            # when the static dir is missing): still revalidate, body as-is.
            response.headers["Cache-Control"] = HTML_CACHE_CONTROL
            return response

        # ``asset_version()`` is lru_cached, so only the very first call in
        # the process can fail — resolve it BEFORE draining the body stream
        # below.
        try:
            token = asset_version()
            body = await _read_body(response)
        except Exception:
            logger.exception(
                "cache busting: could not buffer %s — serving unmodified (no-cache)",
                path,
            )
            response.headers["Cache-Control"] = HTML_CACHE_CONTROL
            return response

        # Phase 91 (task 02): the pre-paint theme tag. One short-lived
        # session per response (the sync-endpoint house pattern from
        # app/db.py — the middleware world is sync); NO process cache —
        # the theme changes at runtime from the admin tab, so the next
        # request must see it without a restart. A DB blip (or a
        # pre-migration boot) must never break the page: fall back to
        # ``tag == ""`` (the built-in palette) and keep the no-cache
        # contract (loadHealth house style).
        tag = ""
        try:
            db = SessionLocal()
            try:
                effective = theming.effective_settings(db)
            finally:
                db.close()
            tag = theming.theme_style_tag(
                {key: effective[key] for key in theming.COLOR_FIELDS}
            )
        except Exception:
            logger.exception(
                "cache busting: theme read failed for %s — serving without the theme tag",
                path,
            )
            tag = ""

        try:
            new_body = theming.inject_theme(
                rewrite_asset_refs(body.decode("utf-8"), token), tag
            ).encode("utf-8")
        except Exception:
            # The body IS buffered — re-serve the ORIGINAL bytes so a
            # rewrite hiccup never loses the page.
            logger.exception("cache busting: rewrite of %s failed — serving raw (no-cache)", path)
            return Response(
                content=body,
                status_code=response.status_code,
                headers=_no_cache_headers(response),
            )

        headers = _no_cache_headers(response)
        if tag:
            # Phase 91 (task 05): the inline tag needs a style-src
            # exemption or the phase-82 CSP blocks it in the browser —
            # the strictest one: a sha256 hash of the EXACT tag content
            # (theming.theme_csp_hash), appended to the A1 string. The
            # outer SecurityHeadersMiddleware preserves this (it only
            # fills in a missing CSP); the untagged page keeps A1
            # verbatim — byte- AND header-identical to pre-phase-91.
            headers["Content-Security-Policy"] = (
                f"{CSP}; style-src 'self' '{theming.theme_csp_hash(tag)}'"
            )
        return Response(
            content=new_body,
            status_code=response.status_code,
            headers=headers,
        )


def configure_caching(app: FastAPI) -> None:
    """Attach :class:`CachingMiddleware` to ``app`` (phase 33).

    Call inside ``create_app()`` after the routers are mounted; the
    middleware wraps the whole app, so the ``/`` static catch-all is
    covered too. ``/api/*`` responses (including the SSE chat stream) are
    never touched by it.
    """
    app.add_middleware(CachingMiddleware)