phase: 188_chunks_hnsw_index
**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
This commit is contained in:
+12
@@ -0,0 +1,12 @@
|
||||
# Shop vacuum
|
||||
|
||||
Dust collection layout for the garage bench and the tools it serves.
|
||||
|
||||
- The cyclone bin empties at the driveway door, twice a week.
|
||||
- Replace the final filter each season or the airflow drops noticeably.
|
||||
- The hose is the flexible 4 inch line, never the rigid conduit.
|
||||
- The saw runs on its own circuit so the motor start does not brown out
|
||||
the bench outlets.
|
||||
|
||||
Airflow is measured at the hood with a cheap gauge; anything under the
|
||||
marker means the cyclone needs emptying before anything else is checked.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Coffee grinder notes
|
||||
|
||||
Burr alignment and dose calibration for the counter grinder. Nothing here
|
||||
applies to the commercial machine at the shop.
|
||||
|
||||
- Grind finer when the shot runs short and sour.
|
||||
- Clean the chute weekly; static holds grounds in the bend.
|
||||
- Recalibrate the dose after any burr change with a blind basket test.
|
||||
- Replace the basket screen when the base of the puck turns muddy.
|
||||
|
||||
The dial clicks are not linear near the fine end, so two clicks of
|
||||
adjustment there behave like five clicks in the middle of the range.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Garden beds
|
||||
|
||||
Crop rotation for the four raised beds behind the shed, following the
|
||||
legume, brassica, root, fruit order so the beds rest every fourth year.
|
||||
|
||||
- Bed one: legumes after the last frost date clears the soil.
|
||||
- Bed two: heavy feeders get a top dressing of compost in spring.
|
||||
- Bed three: roots only, no manure or the carrots fork.
|
||||
- Bed four: fruiting crops staked and watered at the base.
|
||||
|
||||
The irrigation line is one drip tape per bed on a fifteen minute timer,
|
||||
and the mulch goes back on after every harvest rather than once a season.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Kubernetes cluster
|
||||
|
||||
The cluster is three nodes with a control plane on `k8s-master-1` and two
|
||||
workers. It runs the shelf services that want scheduling rather than a
|
||||
static container.
|
||||
|
||||
- The ingress controller listens on 8443 and forwards to cluster services.
|
||||
- etcd snapshots run nightly to the backup host and are kept for a month.
|
||||
- Worker nodes join with a short-lived token minted for one hour.
|
||||
- Storage is a local-path provisioner pinned to the nvme mount.
|
||||
- Pod security labels are set per namespace, not cluster wide.
|
||||
|
||||
Upgrades are one control plane replica at a time, then the workers with a
|
||||
cordon and drain. The ingress is the last thing to move because the
|
||||
services behind it are checked by the shelf watchdog.
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Proxmox nodes
|
||||
|
||||
Three Proxmox hosts form the shelf cluster: node one is the hypervisor and
|
||||
gateway, node two runs the always-on containers, node three is the GPU box.
|
||||
|
||||
- Storage is an LVM-thin pool on the nvme, over-commit watched weekly.
|
||||
- Live migration uses a dedicated bridge, not the management iface.
|
||||
- The watchdog restarts a container that misses three health checks.
|
||||
- Corosync traffic rides the management bridge with multicast.
|
||||
- Backups of the VM configs are taken before every host upgrade.
|
||||
|
||||
Node three is pinned for the inference workloads, so a migration off it is
|
||||
manual and only happens for a scheduled host reboot.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
# Restic backups
|
||||
|
||||
Restic backs up the media volume to the offsite bucket every night at 01:20
|
||||
with a lock timeout of five minutes.
|
||||
|
||||
- The repository password lives in the vault, never in the env file.
|
||||
- Prune keeps fourteen daily and eight weekly snapshots.
|
||||
- Verification runs one read pass every Sunday, sampled rather than full.
|
||||
- A failed lock means the previous run is still going; check before
|
||||
forcing anything.
|
||||
- Restores are tested quarterly against a scratch directory.
|
||||
|
||||
The media volume changes slowly, so a nightly run is usually small and the
|
||||
sampled verification is enough to catch a corrupt pack file early.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
# Ntfy relay
|
||||
|
||||
The ntfy relay runs on `notify-host-1` and fans alert payloads out to the
|
||||
phones and the desktop notifier.
|
||||
|
||||
- Topic namespaces are per host, so a host can only publish into its own.
|
||||
- Retention is twelve hours; anything older is dropped on the sweep.
|
||||
- The access log lives at `/var/log/ntfy/access.log` and rotates weekly.
|
||||
- SMTP fallback is disabled on purpose: a failed push is a signal, not
|
||||
something to paper over with an email.
|
||||
- The relay is fronted by the proxy on 443 with mutual TLS for publishers.
|
||||
|
||||
Publishing is a plain POST with the topic in the path. The relay stamps a
|
||||
monotonic sequence number so a subscriber can detect a gap after a
|
||||
reconnect, and the gap detection is what the shelf watchdog polls.
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
# Tilecache
|
||||
|
||||
Tilecache serves the vector tiles for the map boxes from `map-host-2`,
|
||||
listening on port 8077 behind the reverse proxy. Everything below is the
|
||||
shape of that cache on the shelf, not a general tile server guide.
|
||||
|
||||
## Cache budget and eviction
|
||||
|
||||
The cache budget is 40 GiB on the nvme. The eviction watermark sits at 88
|
||||
percent of that budget: once the cache crosses the watermark, tilecache
|
||||
evicts the coldest tiles until it drops back under it.
|
||||
|
||||
Raise the eviction watermark so the hot zoom levels stay pinned through a
|
||||
bulk re-render:
|
||||
|
||||
1. `tilecache config set eviction-watermark 92`
|
||||
2. `tilecache config set pin-zoom-floor 12`
|
||||
3. `systemctl restart tilecache`
|
||||
|
||||
Zoom levels 12 to 15 are the hot set for the atlas layers, and the pin
|
||||
floor keeps them resident while the watermark holds. Zoom 16 and above is
|
||||
rendered on demand and is always evictable.
|
||||
|
||||
## Origin fetches
|
||||
|
||||
Origin fetches go to `origin.maps.internal`, and a slow origin reply is
|
||||
treated as a miss after 900 ms. A stale origin reply keeps the previous
|
||||
tile pinned rather than evicting the hot zoom set, so a watermark raise is
|
||||
safe during a bulk re-render.
|
||||
|
||||
## Watermark maths
|
||||
|
||||
At a 40 GiB budget the 88 percent watermark means eviction starts at 35.2
|
||||
GiB. The 92 percent setting moves that to 36.8 GiB, which buys roughly one
|
||||
extra render pass of zoom 14 before anything cold is dropped.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
# Wireguard hub
|
||||
|
||||
The Wireguard hub on `edge-host-1` is the only way into the shelf from a
|
||||
laptop. Each device gets one peer entry and one allowed-IPs line.
|
||||
|
||||
- Listen port 51820/udp, one peer per device.
|
||||
- Allowed IPs are per-device /32s; the hub does not advertise the LAN.
|
||||
- Persistent keepalive is 25 seconds for the roaming peers.
|
||||
- Key rotation is a manual ceremony: generate, swap the peer file, reload.
|
||||
- The hub hands out addresses from 10.66.0.0/24, one /32 per peer.
|
||||
|
||||
A peer that roams between the mobile network and wifi must keep the same
|
||||
endpoint or the hub drops the stale mapping after the keepalive window.
|
||||
The routing table is re-read on a reload, so a swapped peer file needs no
|
||||
reboot of the host.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Sourdough timetable
|
||||
|
||||
Bulk fermentation schedule for the weekend bakes at the kitchen bench.
|
||||
|
||||
- Stretch and fold every forty minutes for the first two hours.
|
||||
- Bulk ends when the dough has grown by half and jiggles, not by clock.
|
||||
- Cold retard overnight in the fridge at four degrees.
|
||||
- Bake from cold with steam for the first twenty minutes.
|
||||
|
||||
The starter is fed one to five in the morning so it peaks mid afternoon,
|
||||
which is when the mix actually happens.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Packing list
|
||||
|
||||
Carry-on checklist for a week of trips with one bag and a laptop.
|
||||
|
||||
- One shirt per two days, always folded not rolled.
|
||||
- Spare battery in the personal item, never the checked bag.
|
||||
- Toiletries under the hundred millilitre rule in one clear pouch.
|
||||
- A single pair of shoes that survives both rain and a dinner.
|
||||
|
||||
The laptop bag holds the cables, the drive and the paper documents, and it
|
||||
is the thing that gets opened at every security checkpoint, so it stays
|
||||
boring and easy to repack.
|
||||
Reference in New Issue
Block a user