phase: 188_chunks_hnsw_index
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s

**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
This commit is contained in:
2026-10-17 03:18:23 -04:00
parent c054b47df5
commit f20d40e4fe
81 changed files with 5076 additions and 2 deletions
+12
View File
@@ -0,0 +1,12 @@
# Shop vacuum
Dust collection layout for the garage bench and the tools it serves.
- The cyclone bin empties at the driveway door, twice a week.
- Replace the final filter each season or the airflow drops noticeably.
- The hose is the flexible 4 inch line, never the rigid conduit.
- The saw runs on its own circuit so the motor start does not brown out
the bench outlets.
Airflow is measured at the hood with a cheap gauge; anything under the
marker means the cyclone needs emptying before anything else is checked.
+12
View File
@@ -0,0 +1,12 @@
# Coffee grinder notes
Burr alignment and dose calibration for the counter grinder. Nothing here
applies to the commercial machine at the shop.
- Grind finer when the shot runs short and sour.
- Clean the chute weekly; static holds grounds in the bend.
- Recalibrate the dose after any burr change with a blind basket test.
- Replace the basket screen when the base of the puck turns muddy.
The dial clicks are not linear near the fine end, so two clicks of
adjustment there behave like five clicks in the middle of the range.
+12
View File
@@ -0,0 +1,12 @@
# Garden beds
Crop rotation for the four raised beds behind the shed, following the
legume, brassica, root, fruit order so the beds rest every fourth year.
- Bed one: legumes after the last frost date clears the soil.
- Bed two: heavy feeders get a top dressing of compost in spring.
- Bed three: roots only, no manure or the carrots fork.
- Bed four: fruiting crops staked and watered at the base.
The irrigation line is one drip tape per bed on a fifteen minute timer,
and the mulch goes back on after every harvest rather than once a season.
+15
View File
@@ -0,0 +1,15 @@
# Kubernetes cluster
The cluster is three nodes with a control plane on `k8s-master-1` and two
workers. It runs the shelf services that want scheduling rather than a
static container.
- The ingress controller listens on 8443 and forwards to cluster services.
- etcd snapshots run nightly to the backup host and are kept for a month.
- Worker nodes join with a short-lived token minted for one hour.
- Storage is a local-path provisioner pinned to the nvme mount.
- Pod security labels are set per namespace, not cluster wide.
Upgrades are one control plane replica at a time, then the workers with a
cordon and drain. The ingress is the last thing to move because the
services behind it are checked by the shelf watchdog.
+13
View File
@@ -0,0 +1,13 @@
# Proxmox nodes
Three Proxmox hosts form the shelf cluster: node one is the hypervisor and
gateway, node two runs the always-on containers, node three is the GPU box.
- Storage is an LVM-thin pool on the nvme, over-commit watched weekly.
- Live migration uses a dedicated bridge, not the management iface.
- The watchdog restarts a container that misses three health checks.
- Corosync traffic rides the management bridge with multicast.
- Backups of the VM configs are taken before every host upgrade.
Node three is pinned for the inference workloads, so a migration off it is
manual and only happens for a scheduled host reboot.
+14
View File
@@ -0,0 +1,14 @@
# Restic backups
Restic backs up the media volume to the offsite bucket every night at 01:20
with a lock timeout of five minutes.
- The repository password lives in the vault, never in the env file.
- Prune keeps fourteen daily and eight weekly snapshots.
- Verification runs one read pass every Sunday, sampled rather than full.
- A failed lock means the previous run is still going; check before
forcing anything.
- Restores are tested quarterly against a scratch directory.
The media volume changes slowly, so a nightly run is usually small and the
sampled verification is enough to catch a corrupt pack file early.
+15
View File
@@ -0,0 +1,15 @@
# Ntfy relay
The ntfy relay runs on `notify-host-1` and fans alert payloads out to the
phones and the desktop notifier.
- Topic namespaces are per host, so a host can only publish into its own.
- Retention is twelve hours; anything older is dropped on the sweep.
- The access log lives at `/var/log/ntfy/access.log` and rotates weekly.
- SMTP fallback is disabled on purpose: a failed push is a signal, not
something to paper over with an email.
- The relay is fronted by the proxy on 443 with mutual TLS for publishers.
Publishing is a plain POST with the topic in the path. The relay stamps a
monotonic sequence number so a subscriber can detect a gap after a
reconnect, and the gap detection is what the shelf watchdog polls.
+35
View File
@@ -0,0 +1,35 @@
# Tilecache
Tilecache serves the vector tiles for the map boxes from `map-host-2`,
listening on port 8077 behind the reverse proxy. Everything below is the
shape of that cache on the shelf, not a general tile server guide.
## Cache budget and eviction
The cache budget is 40 GiB on the nvme. The eviction watermark sits at 88
percent of that budget: once the cache crosses the watermark, tilecache
evicts the coldest tiles until it drops back under it.
Raise the eviction watermark so the hot zoom levels stay pinned through a
bulk re-render:
1. `tilecache config set eviction-watermark 92`
2. `tilecache config set pin-zoom-floor 12`
3. `systemctl restart tilecache`
Zoom levels 12 to 15 are the hot set for the atlas layers, and the pin
floor keeps them resident while the watermark holds. Zoom 16 and above is
rendered on demand and is always evictable.
## Origin fetches
Origin fetches go to `origin.maps.internal`, and a slow origin reply is
treated as a miss after 900 ms. A stale origin reply keeps the previous
tile pinned rather than evicting the hot zoom set, so a watermark raise is
safe during a bulk re-render.
## Watermark maths
At a 40 GiB budget the 88 percent watermark means eviction starts at 35.2
GiB. The 92 percent setting moves that to 36.8 GiB, which buys roughly one
extra render pass of zoom 14 before anything cold is dropped.
+15
View File
@@ -0,0 +1,15 @@
# Wireguard hub
The Wireguard hub on `edge-host-1` is the only way into the shelf from a
laptop. Each device gets one peer entry and one allowed-IPs line.
- Listen port 51820/udp, one peer per device.
- Allowed IPs are per-device /32s; the hub does not advertise the LAN.
- Persistent keepalive is 25 seconds for the roaming peers.
- Key rotation is a manual ceremony: generate, swap the peer file, reload.
- The hub hands out addresses from 10.66.0.0/24, one /32 per peer.
A peer that roams between the mobile network and wifi must keep the same
endpoint or the hub drops the stale mapping after the keepalive window.
The routing table is re-read on a reload, so a swapped peer file needs no
reboot of the host.
+11
View File
@@ -0,0 +1,11 @@
# Sourdough timetable
Bulk fermentation schedule for the weekend bakes at the kitchen bench.
- Stretch and fold every forty minutes for the first two hours.
- Bulk ends when the dough has grown by half and jiggles, not by clock.
- Cold retard overnight in the fridge at four degrees.
- Bake from cold with steam for the first twenty minutes.
The starter is fed one to five in the morning so it peaks mid afternoon,
which is when the mix actually happens.
+12
View File
@@ -0,0 +1,12 @@
# Packing list
Carry-on checklist for a week of trips with one bag and a laptop.
- One shirt per two days, always folded not rolled.
- Spare battery in the personal item, never the checked bag.
- Toiletries under the hundred millilitre rule in one clear pouch.
- A single pair of shoes that survives both rain and a dinner.
The laptop bag holds the cables, the drive and the paper documents, and it
is the thing that gets opened at every security checkpoint, so it stays
boring and easy to repack.