ducoterra
7fce6572d0
Single consolidated commit for four completed, validated phases (77, 78, 79, 80). The pipeline run left all work uncommitted because the harness commits only with PHASE_COMMIT=1 while child executors are forbidden from committing; the phases themselves all passed validation and moved to .agents/phases/complete/. Phase 77 — navbar view refresh - router.js dispatches bor:view-refresh on re-show / active re-click / popstate (gated on wasMounted; first show and boot exempt) - History / RAG / Sources / Tuning re-fetch on refresh (admin branch); Chat deliberately excluded (stream survival) - History "Refresh" button (admin-only, in-flight disable + status line) - New story suite tests/e2e/test_navbar_refresh.py (7 tests) Phase 78 — static background - Removed the animated glow layers; static 44px grid over the flat --bg canvas; default and reduced-motion renders byte-identical - Updated background/theme E2E suites; removed bg-glow test pins Phase 79 — API tokens - api_tokens model + migration 0012; hash-only token service - Admin tokens API + Tokens admin view; POST /api/token-auth; live-revoking require_user on chat / suggestions / document content - Frontend token gate with localStorage cache; anonymous E2E suites migrated to token login - New story suite tests/e2e/test_api_tokens.py (9 tests) Phase 80 — history suggestion chips - last_questions() endpoint with SEED fallback; startNewChat() refetch - Seed-semantics docs (config.py, .env.example, README) - Integration state matrix + E2E suite rewritten to the 4 chip states Also included: phase-76 report artifacts and the repo restore-test-db skill (previously untracked), scripts/* ruff fixes from phase 77. Final gate state (phase 80 final pass, covers everything above): - uv run pytest --cov=app → 1637 passed, 0 failed, app/ coverage 99% - uv run ruff check . && uv run pyright → clean, 0 errors - Per-phase story E2E suites green in isolation
135 lines
5.8 KiB
Markdown
135 lines
5.8 KiB
Markdown
---
|
|
name: restore-test-db
|
|
description: Restores the large 1,000-document test knowledge base (the live-KB replica snapshot at data/bor_live_kb_replica.dump.sql) into the app's Postgres database in one atomic transaction — wipes the database's current contents, no re-embedding, no app code changes. Use when the user asks to restore the test database, the 1,000-document KB, or the big/live replica KB (e.g. "restore the test database", "put the 1000-doc KB back"). Do not confuse with the 8-document tool-calling fixture KB, which scripts/restore_test_kb.py restores.
|
|
---
|
|
|
|
# Restore the 1,000-Document Test KB
|
|
|
|
Load `data/bor_live_kb_replica.dump.sql` (a full 99 MB `pg_dump` —
|
|
schema **and** data — of the live KB replica) into the app database
|
|
(`brain_of_reese`, per `BOR_DATABASE_URL` in `.env`). The embeddings come
|
|
with the dump, so there is **no re-embedding, no git clone, no app code
|
|
change** — the whole known state (documents, chunks + embeddings, source
|
|
registry, KB overview, alembic version) lands in a few seconds.
|
|
|
|
The dump is a local artifact: `data/` is gitignored and the dump is
|
|
**never committed**. It is not in git — if it is missing on a fresh
|
|
clone, stop and ask the user for it (it can only be made by `pg_dump`ing
|
|
the live/`bor_eval` KB).
|
|
|
|
## Rules (non-negotiable)
|
|
|
|
- **The restore is destructive** — it wipes the current contents of the
|
|
app database. Always take the disposable backup in step 2 first, even
|
|
when the database looks empty.
|
|
- **Do not edit app code** (`app/`, `tests/`, `alembic/`). This skill
|
|
swaps database contents, nothing else.
|
|
- **Run `psql` inside the db container** (`brain-of-reese_db_1`), not
|
|
with a host `psql` — the container's psql is 17.x, matching the dump
|
|
(newer dumps use the `\restrict` metacommand that old host psqls
|
|
choke on).
|
|
- This KB and the 8-document tool-calling fixture KB
|
|
(`scripts/restore_test_kb.py`) share the same database and overwrite
|
|
each other. Say which one is loaded in the summary.
|
|
|
|
## Procedure
|
|
|
|
All commands run from the repo root.
|
|
|
|
### 1. Preconditions (fast fail)
|
|
|
|
```bash
|
|
podman compose up -d db
|
|
ls -lh data/bor_live_kb_replica.dump.sql # must exist, ~99 MB
|
|
```
|
|
|
|
Check the dump's schema version against the current head:
|
|
|
|
```bash
|
|
grep -A1 '^COPY public.alembic_version' data/bor_live_kb_replica.dump.sql | tail -1
|
|
uv run alembic heads
|
|
```
|
|
|
|
The two must match (the 2026-09-05 snapshot carries `0011`). If the dump
|
|
is **older** than head, proceed and run `uv run alembic upgrade head`
|
|
after step 3. If the dump is **newer** than head, the local code is
|
|
behind — stop and flag it to the user.
|
|
|
|
### 2. Disposable safety backup
|
|
|
|
Seconds, in `/tmp` (never committed):
|
|
|
|
```bash
|
|
podman exec brain-of-reese_db_1 pg_dump -U reese brain_of_reese -Fc \
|
|
> /tmp/pre_restore_$(date +%Y%m%d_%H%M%S).dump
|
|
```
|
|
|
|
### 3. Reset the schema, load the dump (one transaction)
|
|
|
|
```bash
|
|
podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c \
|
|
"DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO PUBLIC;"
|
|
|
|
podman exec -i brain-of-reese_db_1 psql -U reese -d brain_of_reese \
|
|
-v ON_ERROR_STOP=1 --single-transaction -q -f /dev/stdin \
|
|
< data/bor_live_kb_replica.dump.sql
|
|
```
|
|
|
|
Why the schema reset: the dump contains plain `CREATE TABLE` (no
|
|
`IF NOT EXISTS`, no `--clean`), so it can only load into an empty
|
|
`public` schema. Why `--single-transaction` + `ON_ERROR_STOP`: a failed
|
|
load rolls back instead of leaving a half-loaded KB. Known failure mode:
|
|
the `DROP SCHEMA` is its own committed statement, so a failed load leaves
|
|
an **empty** schema — that is fine, fix the cause and re-run step 3
|
|
(the reset is idempotent).
|
|
|
|
### 4. Verify the fingerprint
|
|
|
|
```bash
|
|
podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -tAc \
|
|
"select 'docs='||count(*) from documents
|
|
union all select 'chunks='||count(*) from chunks
|
|
union all select 'sources='||string_agg(distinct source, ',' order by source) from documents
|
|
union all select 'alembic='||(select version_num from alembic_version);"
|
|
```
|
|
|
|
Expected for the 2026-09-05 snapshot: **docs=1000, chunks=8866,
|
|
sources=deploy,homelab,homepage,ServMon,skills,vibe-bot, alembic=0011**.
|
|
If any number differs, the dump file is a different snapshot — report the
|
|
actual numbers to the user; do not guess.
|
|
|
|
### 5. Dev-server check (only if it is running)
|
|
|
|
```bash
|
|
pgrep -af "uvicorn app.main" && curl -s http://localhost:8000/api/health
|
|
```
|
|
|
|
Expect `"db":"up"` — the SQLAlchemy pool reconnects fine across the
|
|
schema reset, so no restart is normally needed. If the server misbehaves
|
|
after the restore, restart it (`uv run uvicorn app.main:app --reload`).
|
|
|
|
### 6. Summary
|
|
|
|
One line per verified number (docs / chunks / sources / alembic), the
|
|
wall time, and a reminder of which KB is loaded: this is the
|
|
**1,000-document live replica**; the 8-document tool-calling fixture is
|
|
brought back separately with `uv run python -m scripts.restore_test_kb`.
|
|
|
|
## Troubleshooting
|
|
|
|
- **`database unreachable` / connection refused** — the db container is
|
|
down; `podman compose up -d db`, then re-run from step 1.
|
|
- **`relation "public.<table>" already exists`** — the schema was not
|
|
reset (or a previous load failed after the reset rolled back); re-run
|
|
the `DROP SCHEMA` command from step 3.
|
|
- **`psql: /dev/stdin:N: unknown command '\restrict'`** — you ran a host
|
|
`psql`; use the container's, per the rules.
|
|
- **load succeeds but counts are wrong** — compare the dump's
|
|
`alembic_version` (step 1) and `documents` row count
|
|
(`sed -n '/^COPY public.documents/,/^\\./p' data/bor_live_kb_replica.dump.sql | wc -l`,
|
|
minus 2) against the loaded numbers; a mismatch means the dump was
|
|
truncated in transfer — ask the user for the original file.
|
|
- **`schema not applied`-style errors from the app afterwards** — run
|
|
`uv run alembic current`; if the dump was older than head, step 1's
|
|
`alembic upgrade head` was the missing piece.
|