Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
brain-of-reese/benchmarks/phase188_live_gate/battery_index_off.txt
T
ducoterra f20d40e4fe
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Details
Build and Push Containers / build-and-push-db (push) Successful in 12s
Details
phase: 188_chunks_hnsw_index 
**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
2026-10-17 03:18:23 -04:00

65 lines
7.0 KiB
Plaintext
Raw Permalink Blame History

eval: threshold=0.62 vector_candidates=100 lexical_candidates=30 rrf_k=60 recency_boost=0.0007 recency_half_life_days=365 name_hit_bonus=0.005
question: 'How do I deploy gitea?'
gate: best_cosine=0.8528 fts_hits=38 -> HIGH (answer)
1. deploy/Deployments/reeseapps/gitea/app/playbook.yaml cosine=0.7640 fts=1 effective=0.03056 created=2026-10-17 (playbook)
2. deploy/Deployments/reeseapps/gateway/README.md cosine=0.7736 fts=1 effective=0.02902 created=2026-10-17 (Proxy Server)
3. homelab/active/container_gitea/gitea.md cosine=0.7612 fts=1 effective=0.02561 created=2026-10-17 (Gitea)
4. homelab/active/kubernetes_gitea/gitea-values.yaml cosine=0.7232 fts=1 effective=0.02512 created=2026-10-17 (gitea-values)
5. deploy/Deployments/reeseapps/gitea/app/gitea-postgres.env.j2 cosine=0.7159 fts=1 effective=0.02495 created=2026-10-17 (gitea-postgres.env)
suggested: 1. deploy/Deployments/reeseapps/gitea/app/playbook.yaml 2. deploy/Deployments/reeseapps/gitea/app/gitea-postgres.env.j2 3. deploy/Deployments/reeseapps/gateway/README.md 4. deploy/Deployments/reeseapps/gitea/README.md 5. deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2
question: 'How do I deploy forgejo?'
gate: best_cosine=0.7876 fts_hits=37 -> HIGH (answer)
1. homelab/templates/container/foobar.md cosine=0.7876 fts=0 effective=0.01709 created=2026-10-17 (Podman foobar)
2. deploy/Deployments/reeseapps/forgejo/README.md cosine=0.0000 fts=1 effective=0.01709 created=2026-10-17 (Reeseapps Forgejo)
3. deploy/Deployments/reeseapps/README.md cosine=0.7857 fts=0 effective=0.01683 created=2026-10-17 (Deployments)
4. deploy/Deployments/reeseapps/forgejo/app/forgejo-app.container cosine=0.0000 fts=1 effective=0.01683 created=2026-10-17 (forgejo-app)
5. homelab/active/container_bifrost/bifrost.md cosine=0.7672 fts=0 effective=0.01657 created=2026-10-17 (Podman bifrost)
suggested: 1. deploy/Deployments/reeseapps/forgejo/README.md 2. deploy/Deployments/reeseapps/forgejo/app/forgejo-app.container 3. deploy/Deployments/reeseapps/forgejo/app/forgejo-app.env.j2 4. deploy/Deployments/reeseapps/forgejo/app/forgejo-postgres.container 5. deploy/Deployments/reeseapps/forgejo/app/forgejo-postgres.env.j2
question: 'How do I set up the gateway?'
gate: best_cosine=0.7816 fts_hits=37 -> HIGH (answer)
1. homelab/active/device_shelly/shelly.md cosine=0.6940 fts=1 effective=0.02509 created=2026-10-17 (Shelly Devices)
2. deploy/Deployments/reeseapps/forgejo/app/forgejo.network cosine=0.6727 fts=1 effective=0.02050 created=2026-10-17 (forgejo)
3. homelab/active/device_framework_16/framework_16.md cosine=0.7816 fts=0 effective=0.01709 created=2026-10-17 (Framework Laptop 16)
4. deploy/Deployments/delta/istio/gateway-metallb.yaml cosine=0.0000 fts=1 effective=0.01709 created=2026-10-17 (gateway-metallb)
5. homelab/active/device_home_assistant/home_assistant.md cosine=0.7752 fts=0 effective=0.01683 created=2026-10-17 (Home Assistant)
suggested: 1. homelab/active/device_shelly/shelly.md 2. deploy/Deployments/delta/istio/gateway-metallb.yaml 3. deploy/Deployments/delta/istio/test-gateway.yaml 4. deploy/Deployments/reeseapps/gateway/README.md 5. deploy/Deployments/reeseapps/gateway/certbot/playbook.yaml
question: 'Do you have notes on the qwen 3.8 27b model?'
gate: best_cosine=0.7483 fts_hits=31 -> HIGH (answer)
1. deploy/Deployments/reeseapps/ai/deployments/epic/quadlets/qwen3.8-27b-epic-vulkan.container cosine=0.7402 fts=1 effective=0.03322 created=2026-10-17 (qwen3.8-27b-epic-vulkan)
2. deploy/Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container cosine=0.7142 fts=1 effective=0.03058 created=2026-10-17 (qwen3.8-27b-juggernaut-vulkan)
3. homelab/active/container_localai/localai.md cosine=0.6837 fts=1 effective=0.02570 created=2026-10-17 (Local AI with Anything LLM)
4. homelab/active/software_ai_stack/ai_stack.md cosine=0.7029 fts=1 effective=0.02545 created=2026-10-17 (Self Hosted AI Stack)
5. deploy/Deployments/reeseapps/ai/deployments/turbo/quadlets/qwen3.8-flash-next.container cosine=0.6675 fts=1 effective=0.02491 created=2026-10-17 (qwen3.8-flash-next)
suggested: 1. deploy/Deployments/reeseapps/ai/deployments/epic/quadlets/qwen3.8-27b-epic-vulkan.container 2. deploy/Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container 3. deploy/Deployments/reeseapps/ai/deployments/turbo/quadlets/qwen3.8-flash-next.container 4. homelab/active/container_localai/localai.md 5. homelab/active/software_ai_stack/ai_stack.md
question: 'What is the capital of Mongolia?'
gate: best_cosine=0.6070 fts_hits=5 -> HIGH (answer)
1. deploy/Deployments/reeseapps/README.md cosine=0.5588 fts=1 effective=0.02382 created=2026-10-17 (Deployments)
2. homelab/active/container_nginx/nginx.md cosine=0.6070 fts=0 effective=0.01709 created=2026-10-17 (Ngnix)
3. homelab/active/container_localai/localai.md cosine=0.0000 fts=1 effective=0.01709 created=2026-10-17 (Local AI with Anything LLM)
4. homelab/active/os_arch/arch-workstation.md cosine=0.6059 fts=0 effective=0.01683 created=2026-10-17 (Workstation)
5. homelab/active/software_k3s/k3s.md cosine=0.0000 fts=1 effective=0.01683 created=2026-10-17 (K3S)
suggested: 1. deploy/Deployments/reeseapps/README.md 2. homelab/active/container_nginx/nginx.md 3. homelab/active/container_localai/localai.md 4. homelab/active/os_arch/arch-workstation.md 5. homelab/active/software_k3s/k3s.md
question: 'How do I build a Besta?'
gate: best_cosine=0.7078 fts_hits=33 -> HIGH (answer)
1. homelab/active/container_bifrost/bifrost.md cosine=0.7078 fts=0 effective=0.01709 created=2026-10-17 (Podman bifrost)
2. homelab/active/software_distoolbox/arch-build.sh cosine=0.0000 fts=1 effective=0.01709 created=2026-10-17 (arch-build)
3. vibe-bot/README.md cosine=0.6962 fts=0 effective=0.01683 created=2026-10-17 (Vibe Discord Bot with RAG Chat History)
4. homepage/build.sh cosine=0.0000 fts=1 effective=0.01683 created=2026-10-17 (build)
5. vibe-bot/build.sh cosine=0.0000 fts=1 effective=0.01657 created=2026-10-17 (build)
suggested: 1. homelab/active/software_distoolbox/arch-build.sh 2. homepage/build.sh 3. vibe-bot/build.sh 4. homelab/active/container_bifrost/bifrost.md 5. vibe-bot/README.md
question: 'How do I build the BESTÅ?'
gate: best_cosine=0.6969 fts_hits=33 -> HIGH (answer)
1. vibe-bot/build.sh cosine=0.6267 fts=1 effective=0.02519 created=2026-10-17 (build)
2. homelab/active/container_bifrost/bifrost.md cosine=0.6969 fts=0 effective=0.01709 created=2026-10-17 (Podman bifrost)
3. homelab/active/software_distoolbox/arch-build.sh cosine=0.0000 fts=1 effective=0.01709 created=2026-10-17 (arch-build)
4. vibe-bot/README.md cosine=0.6887 fts=0 effective=0.01683 created=2026-10-17 (Vibe Discord Bot with RAG Chat History)
5. homepage/build.sh cosine=0.0000 fts=1 effective=0.01683 created=2026-10-17 (build)
suggested: 1. vibe-bot/build.sh 2. homelab/active/software_distoolbox/arch-build.sh 3. homepage/build.sh 4. homelab/active/container_bifrost/bifrost.md 5. vibe-bot/README.md
Reference in New Issue View Git Blame Copy Permalink