ducoterra
8fac4e64d4
The aipi `embed` endpoint was re-pointed to a new backing model and was re-benchmarked over 3 runs. All three gates pass: output stays 768-dim (matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf vectors, and throughput is 35.6 vec/s on the 50-vector probe. Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin tightens and throughput drops ~2.6x, both consistent with a model swap behind the same alias. Vectors are unit-normalized and deterministic. Note: same dimension on a different model means any KB indexed under the old backing model holds stale vectors and needs a re-embed; the local dev DB is empty, so the phase 119/162 retrieval battery was not runnable here. LLM-Generated: true
1.9 KiB
1.9 KiB
| 1 | date | script | model | mode | gate_status | turns | answered | caps | tool_turns | emitted | executed | contract | wall_s | contract_denom | executed_denom | tool_turns_denom | extra1_col | extra1_val | extra2_col | extra2_val |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 2026-09-06 | summary | lite | quality | FAIL | 8 | 8 | 0 | 0 | 0 | 0 | 58 | 26.9 | 58 | 0 | 8 | coherence | 4.0/5 | hallucinations | 4/8 |
| 3 | 2026-09-06 | summary | lite | quality | FAIL | 8 | 8 | 0 | 0 | 0 | 0 | 53 | 25.4 | 53 | 0 | 8 | coherence | 4.0/5 | hallucinations | 5/8 |
| 4 | 2026-09-06 | embed | embed | dimension | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.1 | 1 | 0 | 1 | actual_dim | 768 | expected_dim | 768 |
| 5 | 2026-09-06 | embed | embed | cosine | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.2 | 1 | 0 | 1 | margin | 0.283 | quality | 56 |
| 6 | 2026-09-06 | embed | embed | speed | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.5 | 1 | 0 | 1 | rate | 93.8 vec/s | bad_vectors | 0 |
| 7 | 2026-09-06 | summary | turbo | quality | FAIL | 8 | 8 | 0 | 0 | 0 | 0 | 63 | 25.3 | 63 | 0 | 8 | coherence | 4.0/5 | hallucinations | 3/8 |
| 8 | 2026-09-06 | chat | lite | fixture | PASS | 10 | 10 | 0 | 9 | 9 | 9 | 11 | 40.4 | 12 | 12 | 10 | ||||
| 9 | 2026-09-06 | chat | lite | fixture | PASS | 10 | 10 | 0 | 10 | 9 | 9 | 13 | 40.5 | 14 | 14 | 10 | ||||
| 10 | 2026-09-06 | chat | lite | derived | FAIL | 10 | 10 | 0 | 10 | 5 | 5 | 10 | 38.3 | 14 | 14 | 10 | ||||
| 11 | 2026-09-06 | chat | turbo | fixture | PASS | 10 | 10 | 0 | 7 | 9 | 9 | 9 | 113.7 | 9 | 9 | 10 | ||||
| 12 | 2026-09-06 | chat | turbo | fixture | PASS | 10 | 10 | 0 | 7 | 7 | 7 | 7 | 112.8 | 7 | 7 | 10 | ||||
| 13 | 2026-09-06 | chat | turbo | derived | FAIL | 10 | 10 | 0 | 5 | 5 | 5 | 5 | 97.2 | 5 | 5 | 10 | ||||
| 14 | 2026-10-09 | embed | embed | dimension | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.2 | 1 | 0 | 1 | actual_dim | 768 | expected_dim | 768 |
| 15 | 2026-10-09 | embed | embed | cosine | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.6 | 1 | 0 | 1 | margin | 0.217 | quality | 43 |
| 16 | 2026-10-09 | embed | embed | speed | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1.4 | 1 | 0 | 1 | rate | 35.6 vec/s | bad_vectors | 0 |
| 17 | 2026-10-09 | embed | embed | dimension | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.0 | 1 | 0 | 1 | actual_dim | 768 | expected_dim | 768 |
| 18 | 2026-10-09 | embed | embed | cosine | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.6 | 1 | 0 | 1 | margin | 0.217 | quality | 43 |
| 19 | 2026-10-09 | embed | embed | speed | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1.4 | 1 | 0 | 1 | rate | 35.5 vec/s | bad_vectors | 0 |
| 20 | 2026-10-09 | embed | embed | dimension | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.0 | 1 | 0 | 1 | actual_dim | 768 | expected_dim | 768 |
| 21 | 2026-10-09 | embed | embed | cosine | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.6 | 1 | 0 | 1 | margin | 0.217 | quality | 43 |
| 22 | 2026-10-09 | embed | embed | speed | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1.4 | 1 | 0 | 1 | rate | 35.6 vec/s | bad_vectors | 0 |