brain-of-reese/.agents/phases/complete/79_api_tokens/00_phase.md

Phase 79 — API tokens: admin-issued access to the app (only shared chats stay open)

Source: TODO.md L5 — "Add api tokens that the admin can generate and hand out so people can log in to use the app. The only thing that should be accessible without an API token is shared chats. The web ui should ask for a token before letting a user through and should cache that token in browser storage so they don't have to keep entering it." Story: n/a (TODO-derived — extends the phase-16 single-admin auth, .agents/user_stories/admin-auth.md) Context: app/core/auth.py (the SessionMiddleware cookie session, require_admin, sign_in/sign_out, ADMIN_SESSION_KEY), app/api/auth.py (/api/login, /api/logout, /api/whoami — WhoamiResponse{authenticated, role: "admin"|"anonymous"}), app/models.py (SQLAlchemy 2.0 mapped-column models — SavedChat is the last one) + alembic/versions/ (latest is 0011_doc_drafts.py — the format to mirror), app/api/chat.py (POST /api/chat — public today), app/api/suggestions.py (public today), app/api/docs.py (GET /api/documents/content — the phase-16 soft rule: deliberately public), app/schemas.py (LoginRequest, WhoamiResponse, …), frontend/assets/header.js (the single /api/whoami call site — fetchIsAdmin() returns authenticated === true; the admin-link reveal; the sign-out binding), frontend/index.html (the shell — #app-nav, the #view-* sections, the sign-in/out links, #main), frontend/document.html + frontend/assets/document.js (the document viewer — fetchIsAdmin gates the admin-only edit affordance; the viewer itself is public today), tests/e2e/auth_helpers.py (the real-form login helper), tests/integration/test_auth_api.py (pins the phase-16 contract — "viewer stays public (soft rule) and POST /api/chat still streams" — that soft rule is SUPERSEDED by this phase).

Objective

The admin can generate named API tokens and hand them out; a token holder signs in at the in-app gate and uses the app — chat, suggestion chips, cited documents. The ONLY anonymous content is the shared chats (plus the login/infra endpoints the gate itself needs). Every existing admin-only surface stays admin-only.

Owner decisions (chat, 2026-09-06 — recorded per AGENTS.md rule 3)

• A3 confirmed — token-user scope: a token user (role user) may: POST /api/chat, GET /api/suggestions, GET /api/documents/content, GET /api/whoami, POST /api/logout. Admin-only UNCHANGED: the docs list/import/sync, tuning, git sources, doc drafts, and the saved-chats list/save/share/delete (saved chats have no per-user attribution — token users get NO History view; only the admin sees saved chats).
• A4 confirmed — token shape & lifecycle: bor_ + 32 hex chars (secrets.token_hex(16)); only the SHA-256 hex digest of the FULL token string is stored (unique index) — the plaintext is returned EXACTLY ONCE at creation. revoked_at set = dead, and revocation is enforced IMMEDIATELY on the user's next request (the session stores the token id; require_user live-checks the row is unrevoked — no server-side session store is added, just a PK lookup).
• A5 confirmed — the gate: an in-app token-entry overlay on the shell + the same inline gate on document.html; login.html (admin password) and shared.html (anonymous) are UNCHANGED. The entered token is cached in localStorage["bor.token"] and silently re-sent to POST /api/token-auth on every page load (a failed silent re-auth — revoked token — drops the key and shows the gate). Sign out clears the key. /api/config stays public (the gate UI itself needs the branding).
• Auth error semantics: an unauthenticated (or revoked) caller to a require_user endpoint gets 401 {"detail": "authentication required"} — 401, not 403 (there is no higher privilege that would unblock them); require_admin keeps its 403 admin only. POST /api/token-auth failures (malformed / unknown / revoked) all get ONE generic 401 {"detail": "invalid token"} (no enumeration — the phase-16 pattern).
• whoami shape: WhoamiResponse{authenticated: bool, role: "admin"|"user"|"anonymous"} — authenticated is true for admin AND user; ALL UI gating switches from authenticated to role === "admin" (the frontend change is owned by task 05). An admin-signed-in session keeps working exactly as today (a browser that holds BOTH an admin and a user session reports admin; sign_out clears everything — one session dict, one logout).

Design (shared by all tasks — the executor reads this, not the chat)

• Model — api_tokens (migration 0012_api_tokens.py): id UUID PK (uuid4 default); label String(120) NOT NULL (the hand-out name, e.g. "alice" — display-only, no index, not unique); token_hash String(64) NOT NULL UNIQUE (the sha256 hex digest of the full bor_… string — the documents.content_hash String(64) precedent); created_at TIMESTAMPTZ NOT NULL server-default now; last_used_at TIMESTAMPTZ NULL; revoked_at TIMESTAMPTZ NULL.
• Service — app/core/tokens.py (new): generate_token() -> str ("bor_" + secrets.token_hex(16)); hash_token(token) -> str (sha256 hexdigest of the FULL token — hashing the full string, not the suffix, so a stripped prefix can never collide); create_token(db, label) -> tuple[ApiToken, str] (returns the row + the plaintext exactly once — the row only ever carries the hash); find_active_by_token(db, token) -> ApiToken | None (hash → token_hash == lookup → revoked_at IS NULL); mark_used(tok) (bump last_used_at to now — the caller commits); revoke(db, token_id) -> bool (set revoked_at when not already — False when the row is missing). Module docstring: the lookup is by HASH (a unique-index hit) — sha256's pre-image resistance means there is no token-enumeration or timing surface beyond the DB lookup (the contrast with check_password's constant-time compare is documented, not replicated — there is nothing to compare in constant time here, only to look up).
• Admin API — app/api/tokens.py (new router, tags=["tokens"], router-level dependencies=[Depends(require_admin)] — the doc_drafts.py pattern): POST /tokens body TokenCreateRequest{label} → 201 TokenCreated{id, label, token, created_at} — the ONLY response that ever carries the plaintext; GET /tokens → TokenList{tokens: [TokenListItem{id, label, created_at, last_used_at, revoked: bool}]} newest-first (no hashes, no plaintext); POST /tokens/{id}/revoke → 204, idempotent (already-revoked → still 204; unknown id → 404 token not found). Registered in app/main.py with the other API routers (before the static mount).
• Auth API — app/api/auth.py: new POST /token-auth (PUBLIC — it is the login): body TokenAuthRequest{token} → find_active_by_token → miss → 401 invalid token; hit → mark_used + commit + session[USER_SESSION_KEY] = True + session[USER_TOKEN_ID_KEY] = str(token.id) → 204. whoami reports the three roles. logout is unchanged (its session.clear() already wipes both roles).
• require_user (in app/core/auth.py): def require_user(request: Request, db: Session = Depends(get_db)) — admin key set → pass; user key set → fetch the ApiToken row by user_token_id (PK hit) — row missing OR revoked_at set → pop BOTH user keys from the session + raise 401 authentication required; else pass; neither key → 401 same detail. Applied to exactly three endpoints: POST /api/chat (app/api/chat.py), GET /api/suggestions (app/api/suggestions.py), GET /api/documents/content (app/api/docs.py — update its docstring: the phase-16 "deliberately PUBLIC soft rule" is SUPERSEDED — the shared chats page is now the anonymous surface). Everything else: unchanged.
• Public list (the only anonymous access — the owner's sentence): /api/health, /api/config, /api/whoami, /api/login, /api/token-auth, /api/shared/<token> (JSON snapshot) + the /shared/<token> page + shared.html, the static assets, and the page documents themselves (login.html, document.html, the shell — the documents load; their GATED DATA does not: the shell shows the gate, document.html shows its inline gate).
• Frontend gate (task 05): new frontend/assets/token-gate.js (module) exposing mountGate(lockRoot, onAuthed): at call — (1) if localStorage["bor.token"] exists → POST /api/token-auth with it (silent; on failure remove the key — it may have been revoked — and fall through); (2) fetchWhoami() → user or admin → onAuthed() (the gate never shows); anonymous → show the gate AND lockRoot.inert = true (the shell passes #main; document.html passes its content wrapper) + focus the token input. Submit → token-auth → 204 → localStorage.setItem("bor.token", …) → whoami → user → hide the gate (hidden + inert on the gate — the ship-hidden pattern), lockRoot.inert = false, onAuthed(). 401 → #auth-gate-error (role="alert") visible, input cleared + re-focused. All localStorage access in try/catch (private mode → the gate still works, caching is a no-op — the fail-silence storage contract). header.js: the single whoami now caches the FULL {authenticated, role} in one module promise (fetchWhoami()); fetchIsAdmin() becomes fetchWhoami().then(w => w.role === "admin") — SAME single request, all existing callers keep working; initSharedHeader() switches its admin variable to role === "admin" (byte-identical behavior for admin/anonymous; a user gets: sign-in hidden, sign-out visible, all admin nav links hidden, steering panel removed — the anonymous branch); the sign-out binding gains localStorage.removeItem("bor.token") (try/catch, before the reload).
• Gate markup (shell — index.html): body-level <section class="auth-gate" id="auth-gate" hidden inert aria-labelledby="auth-gate-title"> AFTER #main (a position: fixed; inset: 0 overlay — the body-level doc-modal precedent): the #sources-gate visual language (glyph, h2 #auth-gate-title "Enter your access token", sub-text pointing at the admin, a <form id="auth-gate-form"> with a visually-hidden label + <input id="auth-gate-input" type="text" autocomplete="off" autocapitalize="none" spellcheck="false" required> (mono), a [Sign in] submit, #auth-gate-error (role="alert", hidden), and a "Sign in as admin" link to /login.html?next=/ (the header's ?next= convention). document.html carries the same markup as #doc-auth-gate (task 05).
• Tokens admin view (task 06): a sixth navbar view folded per the phase-76 pattern — #nav-tokens (ships hidden; header.js reveals it for role admin, same contract as the other four links), #view-tokens section in index.html, new frontend/assets/tokens.js (export async function mount(root), admin-gated via fetchIsAdmin() like history.js), router.js entries in VIEW / VIEW_PATH / VIEW_MODULES / TITLES / DESCRIPTIONS (the brand-composition replaceAll contract carries over), "/tokens.html" in BOTH app/main.py's _shell_routes tuple and app/core/caching.py's HTML_PAGES (the no-cache + ?v= contract — the phase-76 task-03 integration-test updates apply: the shell-route / title-table / _page_file override assertions gain the path). UI: a create row (label input + [Generate]) → the plaintext appears ONCE in a mono read-only field + [Copy] (the clipboard + inline-fallback house pattern — tokens.js keeps its own ~10-line copy, the per-page duplication house style); the once-block is NOT re-shown on a re-render/re-show (the plaintext is gone); a full-width table Label | Created | Last used | Status (Active em-dash vs rose Revoked pill — the stale-pill visual language) | Actions (Revoke — the inline two-step confirm, the history-confirm-* pattern, focus to Yes); a role="status" live region.
• E2E migration (task 04): ten chat suites POST to /api/chat anonymously today and must sign in first (auth_helpers.login(page, app_url, next="/")): test_agent_document_tools.py, test_agent_unlimited_tools.py, test_chat_rag.py, test_grep_regex_teaching.py, test_harness_aligned_tools.py, test_honest_deflection.py, test_llm_retry.py, test_search_tool.py, test_tool_path_teaching.py, test_tool_scaffolding_guardrails.py. auth_helpers.py gains login_with_token(page, app_url, token) — it drives the REAL gate (fill #auth-gate-input → submit → wait for the gate to hide); tests/e2e/test_admin_auth.py's anonymous pins that the app is open (chat streams, viewer public) are updated to the 401/gate contract (its password-flow assertions stay). The shared-chat suites stay ANONYMOUS — that is the point of the item.
• Integration test updates (task 03): the tests that hit the three gated endpoints anonymously (tests/integration/test_api.py, test_chat_api.py, test_auth_api.py, …) sign in as admin first or assert the new 401 where the test's purpose IS the auth contract.

Dependencies

— (none; extends the completed phase-16 auth; phase 80 builds on this phase's /api/suggestions gating)

Tasks

1. 01_token_model_migration.md — the api_tokens model + migration 0012_api_tokens.py.
2. 02_token_admin_api.md — the token service + the admin create/list/revoke endpoints.
3. 03_token_auth_enforcement.md — POST /api/token-auth, the three-role whoami, require_user (live revoke check), enforcement on chat/suggestions/document-content, the integration-contract updates.
4. 04_migrate_anonymous_e2e.md — the login_with_token helper + the ten anonymous chat suites sign in; the E2E inventory is green against the gated app.
5. 05_frontend_token_gate.md — the header role plumbing + the shell gate + the localStorage caching + the document.html gate.
6. 06_tokens_admin_view.md — the admin Tokens view (phase-76 fold pattern) with generate / list / revoke.
7. 07_e2e_story_suite.md — tests/e2e/test_api_tokens.py — the owner's sentence, pinned in a browser.
8. 08_regression_sweep_commit.md — the full pipeline + the README auth section + the atomic commit.

Testing & Quality

• Unit: tests/unit/test_tokens.py (the service — shape, hash, create/find round-trip, revocation, last-used, the malformed/unknown/revoked miss paths) + tests/unit/test_auth.py extended (the require_user matrix: admin pass, active user pass, revoked user 401 + session keys popped, missing row 401, anonymous 401).
• Integration: the admin API (201 plaintext-once, list shape without hashes, revoke idempotency, 403 anonymous, 403 token-user); token-auth (valid / invalid / revoked / malformed); the enforcement matrix on the three endpoints; whoami's three roles; logout clearing the token session; the existing anonymous-chat pins updated to the 401 contract.
• E2E: the new story suite (task 07) + the migrated suites (task 04) + test_admin_auth.py updated + the shared-chat suites green ANONYMOUS.
• Coverage: >90% on app/ (the delta: app/core/tokens.py, app/api/tokens.py, and the modified auth/chat/suggestions/docs files — every new branch tested).

Completion Criteria

• uv run pytest tests/e2e/test_api_tokens.py -v --no-cov green in isolation.
• A token user (fresh browser context) can chat end-to-end (mock LLM) and open a cited document; an anonymous caller gets the gate in the UI and 401s on the API; shared chats open anonymously; every admin surface 403s the token user.
• The cached token survives a reload with no re-entry; sign out clears it; a revoked token is refused on the next request AND on a fresh login attempt.
• Full suite green, coverage >90%, uv run ruff check . && uv run pyright clean; one atomic --no-gpg-sign commit; phase dir moved to .agents/phases/complete/.