Files
brain-of-reese/app/schemas.py
T
ducoterra 03d26255c6 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
2026-08-28 15:57:59 -04:00

285 lines
8.7 KiB
Python

"""Pydantic request/response schemas (API contract)."""
from __future__ import annotations
import uuid
from datetime import datetime
from typing import Literal
from pydantic import BaseModel, Field, field_validator
class HealthResponse(BaseModel):
status: str
db: str
version: str
environment: str
class SuggestionList(BaseModel):
suggestions: list[str]
class ChatRequest(BaseModel):
message: str = Field(min_length=1, max_length=4000)
class LoginRequest(BaseModel):
"""``POST /api/login`` body (phase 16): the single admin's password.
An empty or wrong password is a 401 with one generic detail — never a
422 that would hint at input-shape differences.
"""
password: str = ""
class WhoamiResponse(BaseModel):
"""``GET /api/whoami`` (phase 16) — drives all UI gating."""
authenticated: bool
role: str # "admin" | "anonymous"
class SourceRef(BaseModel):
source: str
path: str
title: str
class ChatThinkingEvent(BaseModel):
"""SSE thinking event: one chunk of the model's reasoning (phase 17).
PLAN §4 extension (A15, owner permission 2026-08-23): frames of the
shape ``{type: "thinking", text: str}`` stream ahead of the
``delta`` frames in practice (the model reasons before it answers). The
client renders them in a collapsible "Thinking" block; the ``done``
event shape is unchanged and thinking text never travels on it.
Sibling of :class:`ChatErrorEvent`.
"""
type: str = "thinking"
text: str
class ChatToolEvent(BaseModel):
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension).
A15 extension (owner permission 2026-08-26): a grounded turn may call
the server-side document tools (``list_documents`` / ``read_document``,
see :mod:`app.rag.agent`); each model-requested call streams as
``{type: "tool", name: str, argument: str | null}`` ahead of the
answer's ``delta`` frames. ``argument`` is the read document's
``"source/path"`` for ``read_document`` and null otherwise. The client
renders each frame as a "calling tool" line/state (phase 37 task 05);
the ``delta`` / ``done`` shapes are unchanged — the read document is
reflected in ``done.sources`` instead.
"""
type: Literal["tool"] = "tool"
name: str # "list_documents" | "read_document"
argument: str | None = None # "source/path" for read_document
class ChatDoneEvent(BaseModel):
"""Final SSE event of a chat turn: metadata for the finished answer."""
type: str = "done"
deflected: bool
sources: list[SourceRef]
suggestions: list[str] = []
class ChatErrorEvent(BaseModel):
"""SSE error event: a turn that cannot complete (PLAN §4).
The client's loading-feedback state machine (phase 06) keys off this
exact shape — ``{type: "error", detail: str}`` — to flip to the error
state and re-enable the send button.
"""
type: str = "error"
detail: str
class DocSummary(BaseModel):
"""One indexed document as shown on the Sources page / API."""
id: str
source: str
path: str
title: str
chunks: int
indexed_at: str
class DocList(BaseModel):
"""Response of ``GET /api/docs`` (empty list → designed empty state)."""
documents: list[DocSummary]
class DocContent(BaseModel):
"""One indexed document's full content (feeds the viewer page, phase 10)."""
source: str
path: str
title: str
format: str
#: Lite-model summary (phase 30) — non-markdown A9 docs only; None for
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
#: summary generation failed but the document was still indexed.
summary: str | None = None
content: str
indexed_at: str
chunks: int
class SteeringNoteIn(BaseModel):
"""``POST /api/steering`` body: one tuning instruction (phase 15).
The note is trimmed *before* the length constraints run, so a
whitespace-only body is a 422 and a 2000-char note with surrounding
spaces still passes.
"""
note: str = Field(min_length=1, max_length=2000)
@field_validator("note", mode="before")
@classmethod
def _trim_note(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
class SteeringNoteUpdate(BaseModel):
"""``PUT /api/steering/{id}`` body: a new tuning instruction (phase 27).
Mirrors :class:`SteeringNoteIn` — the note is trimmed *before* the
length constraints run, so an empty/whitespace body is a 422 and a
full replacement that is ≤2000 chars after the trim still passes.
"""
note: str = Field(min_length=1, max_length=2000)
@field_validator("note", mode="before")
@classmethod
def _trim_note(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
class SteeringNote(BaseModel):
"""One stored steering note (API shape — ISO-8601 ``created_at``)."""
id: uuid.UUID
note: str
created_at: datetime
class SteeringNoteList(BaseModel):
"""``GET /api/steering`` response: all notes, newest first."""
notes: list[SteeringNote]
class GitSourceIn(BaseModel):
"""``POST /api/git-sources`` body (phase 35, task 02; ``kind``, phase 38).
``kind`` selects the source kind and which field carries its location:
* ``"git"`` (default) — ``url`` is the repo URL. Mirrors the
phase-35 contract: trimmed *before* the length constraints run, so a
whitespace-only body is a 422 and a URL with surrounding spaces is
stored clean. Shape validation (``https://``, ``ssh://``, ``git@``)
and the kind-field rules (url present, no path) happen in the API
layer so the 422/409 details stay fixed strings that never echo the
input (credential safety).
* ``"local"`` — ``path`` is an existing directory on the server.
Trimmed here; the API layer then ``expanduser()``s it and requires an
absolute existing directory (else 422 naming the path — the path is
not a secret, unlike a git URL) and no ``url``.
"""
kind: Literal["git", "local"] = "git"
url: str | None = Field(default=None, min_length=1, max_length=500)
path: str | None = Field(default=None, min_length=1, max_length=2000)
@field_validator("url", mode="before")
@classmethod
def _trim_url(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
@field_validator("path", mode="before")
@classmethod
def _trim_path(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
class GitSourceOut(BaseModel):
"""One created git source as returned by ``POST`` (phase 35, task 02).
``id`` / ``added_at`` are non-null for a stored row. ``url`` is the
row's location column: the repo URL for ``kind=git`` rows and, for
``kind=local`` rows, the stored (expanded) directory path — the
phase-35 response shape is unchanged by phase 38, so a local 201
reports its path in ``url`` and the full row (``kind`` + ``path``)
via ``GET``.
"""
id: uuid.UUID | None
url: str
added_at: datetime | None
class GitSourceRow(BaseModel):
"""One row of ``GET /api/git-sources`` (phase 35; ``kind``/``path``,
phase 38, task 02).
``kind`` discriminates the row: git rows (and the git-only
``BOR_GIT_SOURCES`` env-fallback rows) carry ``url`` and
``path: null``; local rows carry ``path`` (the absolute directory,
expanded) and the same string in ``url`` (the table's NOT-NULL
location column). ``id`` / ``added_at`` are nullable: env-fallback
rows (table empty) carry neither.
"""
id: uuid.UUID | None
kind: Literal["git", "local"]
url: str
path: str | None
added_at: datetime | None
class GitSourceList(BaseModel):
"""``GET /api/git-sources`` response (phase 35, task 02).
``from_env`` is True only when the ``git_sources`` table is empty and
the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback
— env rows are git-only and report ``kind: "git"``, ``path: null``);
once the table has rows the env var is ignored and ``from_env`` is
False — the UI is the source of truth.
"""
sources: list[GitSourceRow]
from_env: bool
class UploadOut(BaseModel):
"""``POST /api/git-sources/upload`` response (phase 49, task 02).
The uploaded source's name (filename minus the archive suffix) plus
the SAME count keys as the admin sync's success ``detail``
(``files``, ``added``, ``updated``, ``unchanged``, ``pruned``,
``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the
``overview`` flag: the Sources page renders the same
"N added · N pruned" result line for both.
"""
source: str
files: int
added: int
updated: int
unchanged: int
pruned: int
errors: int
chunks: int
overview: bool