Commit Graph
11 Commits
Author SHA1 Message Date
ducoterra 88293ed02f feat(rag): retry a failed LLM request before the first token lands — BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY with a live 'retrying' status 2026-09-02 10:52:38 -04:00
ducoterra 5d679f5184 feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
2026-08-28 07:02:24 -04:00
ducoterra b855d0aef9 feat(rag): unbounded agent tool calls behind a round cap (owner revision)
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:

- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
  negative rejected) replaces agent_list_calls / agent_read_calls;
  .env.example + README document the single knob; app/rag/prompts.py
  docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
  rounds >= max_rounds, at which point it forces one final no-tools
  answer (the cap is the only forced exit); 0 = no tools — exactly one
  tools=None request, byte-identical to the pre-phase-37 path (the
  kill switch). Rejected calls (unknown tool / missing args /
  already-in-context / unknown path) still consume a round, so
  pathological rejected-call streams are bounded by the cap. The
  per-call log line is now tool/args/round=N/M; the per-turn
  tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
  the deterministic list -> read #1 -> read #2 -> forced-answer flow
  (byte-stable "I read <sp1> and <sp2>." line), classified by the
  count of tool-role read results; the phase-37 single-read flow stays
  byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
  mock-only): three tool frames/lines in order (one list, two reads —
  the second read is what the old read budget refused) + the
  both-named non-deflected answer; done.sources + chips = retrieval
  doc + both reads, deduped; no budget refusal rendered; the
  single-read marker flow regression (exactly one read, single tool
  pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
  the only PLAN edit this phase; the phase-37 note's budget clause is
  marked removed.

Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.

uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.

Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
2026-08-28 04:50:56 -04:00
ducoterra bc70ce36e0 feat(chat): render markdown tables in answers, viewer, and thinking
GFM pipe tables in the shared renderer (TODO.md L6): a table-protection
pass in frontend/assets/markdown.js (fences -> tables -> escape order)
pulls each header+separator+body block out as a placeholder, renders
cells escape-first with the same inline transforms, and reinserts a
semantic <table class="md-table"> inside a horizontal-overflow
.md-table-wrap — so a pipe table in a chat answer, the document
viewer/modal, and the thinking block all render the same semantic
table. Fences win over tables; lone pipes stay text.

- styles.css: .md-table palette rules (PLAN §7.2 tokens, no motion);
  min-width: max-content so a WIDE table keeps its natural width and
  the wrapper is the real scroller (width:100% alone wrapped the wide
  table's cells — proven by the new E2E).
- mock_llm.py: TABLE_TRIGGER ("show me a table") -> byte-stable
  TABLE_ANSWER (3-column table, <img onerror> XSS probe line, wide
  5-column table), checked before DEFLECT_MODE like SUMMARY_MODE.
- tests/fixtures/docs/homelab/tables.md: 3x3 pipe table + pipe-heavy
  fenced block (viewer/fence subject); the shared fixture set grows
  8 -> 9 docs, so every suite pinning the count (added/formats/
  stat-docs/EXPECTED_ROWS) is updated accordingly.
- tests/e2e/test_markdown_tables.py (new, story suite): chat table
  shape + non-deflection, wide-table wrapper scroll (no page
  overflow), XSS probe inert, viewer modal table, fence-not-a-table,
  lone pipe stays text.
- tests/e2e/test_agent_document_tools.py: fix a pre-existing flake —
  the "Calling tool…" label window is ~0.4 s at the mock's 0.1 s
  tool-frame pacing, and a polling expect could stride over it
  (failed 3 of 5 runs on the committed baseline). The pre-submit
  MutationObserver record is the deterministic source of truth; the
  racy to_have_text gate is gone.

uv run pytest: 738 passed, app/ coverage 99% (TOTAL unchanged);
ruff + pyright clean; story E2E 6/6 in isolation; regression E2E
suites (chat_rag, document_viewer, document_summaries, smoke) green.
2026-08-28 03:35:50 -04:00
ducoterra 15c1272828 feat(rag): agent document tools — list/read tools with env-tuned budgets, SSE tool events + "calling tool" UI
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).

SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).

E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).

probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
2026-08-26 22:39:14 -04:00
ducoterra 572a4190a6 feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc 2026-08-25 17:48:37 -04:00
ducoterra b16deb2b1d feat(chat): stream model thinking over SSE and show it in a collapsible block 2026-08-24 09:52:27 -04:00
ducoterra 7e8d14702e feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document 2026-08-22 01:27:02 -04:00
ducoterra e5810b0bcf feat(ui): explicit chat state machine — typing indicator, streaming progress, timeout and error recovery 2026-08-21 18:45:27 -04:00
ducoterra cbf8e39e63 feat(rag): honest deflection gate with amber UI state and alternative-question chips 2026-08-21 17:50:33 -04:00
ducoterra 396e4d47fb feat(rag): stream grounded RAG answers over SSE with source citations
Phase 03 (Story: Chat RAG Answer — happy path):
- app/rag/retriever.py: top-k cosine search + parent-doc selection with
  per-doc dedupe and BOR_MAX_CONTEXT_CHARS cap ([…truncated…] marker)
- app/rag/prompts.py: locked persona + HIGH/DEFLECT prompt builders
- app/rag/llm.py: LLMError + chat_stream (turbo, temp 0.4, max 700, stream)
- app/api/chat.py: POST /api/chat SSE — delta* then done{deflected,
  sources, suggestions}; query_log row + PLAN §9 per-turn log line;
  structured error event on mid-stream failure, JSON 503 when DB down
- frontend: SSE reader, live bubble streaming, source chips -> /sources.html,
  red role=alert banner, Send button state that always recovers
- fix(scaffold): [hidden] { display: none !important } — .kb-banner's
  display:flex was overriding the hidden attribute (banner always visible)
- tests: unit (retriever/prompts/sse/llm) + integration (real Postgres RAG
  turn, query_log, error + 503 paths, mid-turn failures) + Playwright story
  suite (grounded answer, log row, raw SSE shape); smoke placeholder test
  replaced with the real never-stale-button contract
2026-08-21 17:17:02 -04:00