Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 125_image_anchor_context
Build and Push Containers / build-and-push-app (push) Successful in 2m8s
Details
Build and Push Containers / build-and-push-db (push) Failing after 1m38s
Details

Browse Source 
All completion criteria verified. The repository is left functional with all changes in the working tree (no commits made, per the harness protocol).

## Final report — phase 125 `image_anchor_context` (verification pass)

All 6 tasks were already complete; this pass verified every criterion and ran the full gates:

- **Migration 0023** `documents.image_context` (TEXT nullable, reversible) — applied (`alembic current` = 0023 head); round-trip test (4 passed) exercises real upgrade → downgrade 0022 → upgrade.
- **Criterion 1** (referenced image enriched + context line in retrieval): integration `test_markdown_referenced_image_gets_context_and_enriched_chunks` + E2E `test_sync_enriches_the_image_doc_with_markdown_context`, `test_chat_answer_quotes_the_context_line` (wire prompt pinned: `<document>` block leads with the one-line `Referenced in …: "alt"` line) — verified present and passing.
- **Criterion 2** (standalone image byte-identical): E2E `test_standalone_*` (NULL `image_context`, block body exactly the bare description, no `Referenced in` anywhere in the prompt) + unit `test_describe_image_without_context_is_byte_identical` — passing.
- **Criterion 3** (markdown-first: record stored on markdown doc, image picks up context later): integration `test_markdown_indexed_before_image_stores_record_and_image_picks_it_up` + post-index enrichment pass test (markdown file gone from disk) — passing.
- **Gates:** `uv run pytest` → 2938 passed, exit 0 · `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings.
- **Phase E2E in isolation:** `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov` → 4 passed.
- **Mock-LLM change regression check:** neighboring image E2E suites green in isolation — `test_smoke` (3), `test_image_documents` (16), `test_chat_image_questions` (17), `test_answer_images` (17).
- No defects found; no code changes needed. No live-infrastructure changes (local dev DB only). No commit made (harness commits).
- Next pending phase: check `.agents/phases/todo/` after the harness moves this phase.
This commit is contained in:
ducoterra
2026-09-27 05:53:12 -04:00
parent 62e4f6b572
commit e0307fe3ff
60 changed files with 6583 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/importer.py
+592 -64
View File
@@ -85,6 +85,40 @@ summary (stored verbatim, no ``lite`` call, no pointer line), so the
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
equals ``Document.content``.
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
description, the same source's importable markdown files are scanned for
``![alt](path)`` references that name the image (the task-02
:func:`app.rag.md_images.extract_image_refs`); when found, the combined
context string (``Referenced in <source>/<md>:`` + alt text + surrounding
paragraphs, see :func:`_markdown_image_context`) is (a) passed to the
vision call as ``context`` (appended AFTER the locked describe prompt —
the model learns what the image is for) and (b) stored on the doc's
``image_context`` (the retrieval display's context line, task 05), and
the stored ``content`` is ``<context>\n\n<description>`` (context first
for retrieval; ``summary`` mirrors it). A standalone image (no markdown
reference) gets ``image_context = NULL`` and the bare description —
byte-identical to pre-phase-125.
Markdown-FIRST image context (phase 125, task 04, LOCKED A1): a
markdown file (re)indexed while a reference's image has NO document in
the source yet records those references on ITS OWN ``image_context``
(the ``IMAGE_CONTEXT:`` record — :func:`_format_missing_image_refs`;
references are resolved exactly as the task-03 scan, and a reference
escaping the source is not recorded). A reference that DOES name an
existing image doc records nothing (the task-03 path owns that case —
enrichment at image-index time — and the markdown doc's field stays
NULL). After ALL of a source's files are indexed, the post-index
enrichment pass (:func:`_enrich_pending_images`) enriches the still-
``image_context IS NULL`` image docs that a markdown doc's record
references — the case where the markdown was indexed first and its
on-disk file is gone by the time the image is indexed (the task-03
disk scan cannot see it) — with the task-03 mechanics (re-describe
with context, ``content = <context>\n\n<description>``, the context
stored, re-chunk + re-embed). The pass is idempotent (an image doc
with ``image_context`` set is never touched; a markdown doc's record
is NOT cleared on success — it is the retrieval display's record
also, task 05).
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
"""
@@ -92,11 +126,13 @@ from __future__ import annotations
import hashlib
import logging
import posixpath
import re
import uuid
from collections.abc import Callable
from collections.abc import Callable, Sequence
from dataclasses import dataclass, field
from datetime import UTC, datetime
from pathlib import Path
from pathlib import Path, PurePosixPath
from typing import Any, Protocol
from sqlalchemy import select
@@ -108,6 +144,7 @@ from app.models import Chunk, Document
from app.rag.chunker import chunk_document, extract_title
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
from app.rag.llm import EmbeddingError, LLMError
from app.rag.md_images import ImageRef, extract_image_refs
from app.rag.summarizer import (
IMAGE_FALLBACK_MIME,
IMAGE_MIMES,
@@ -122,6 +159,12 @@ EXCLUDED_DIRS: frozenset[str] = frozenset(
{".venv", "node_modules", ".git", "__pycache__", ".pytest_cache", "dist", "build"}
)
#: Markdown extensions that can carry ``![alt](path)`` image references
#: (phase 125, task 03) — the two markdown tokens of the A9 family. The
#: scan (see :func:`_markdown_image_context`) only reads these, and only
#: among the source's importable files.
_MARKDOWN_EXTENSIONS: frozenset[str] = frozenset({".md", ".markdown"})
class Embedder(Protocol):
"""Everything the importer needs from the LLM client (duck-typed for tests)."""
@@ -409,6 +452,18 @@ async def import_sources(
file normally (row + ``image_dir`` copy); with the toggle OFF the
prune skips ``is_image`` docs (the prune guard — the image is
invisible to the walk, not a deleted file).
Markdown-FIRST image context (phase 125, task 04): after ALL of a
source's files are indexed, the post-index enrichment pass
(:func:`_enrich_pending_images`) runs for that source — it enriches
the still-unenriched image docs (``image_context IS NULL``) that a
markdown doc's ``IMAGE_CONTEXT:`` record references (the case where
the markdown was indexed first and its on-disk file is gone by the
time the image is indexed — the task-03 disk scan cannot see it).
The pass is driven by committed doc rows, so it is safe under
``limit`` too (it deletes nothing), and a per-image failure
(missing copy, failed re-description, embed error) keeps that image
as-is and moves on.
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -496,6 +551,14 @@ async def import_sources(
session.rollback()
summary.errors += 1
logger.error("import: error source=%s path=%s — %s", source, rel, e)
# Phase 125 (task 04): the post-index enrichment pass — the
# markdown-FIRST case (a markdown indexed while its
# referenced image had no document yet recorded the
# reference on its doc row; the image was just indexed, but
# the task-03 disk scan could not see the reference — the
# markdown file is no longer on disk). Idempotent +
# fail-soft (see the function).
await _enrich_pending_images(session, source=source, llm=llm, summary=summary)
if prune:
if limit is not None:
logger.warning("import: --prune ignored because --limit was given")
@@ -511,6 +574,50 @@ async def import_sources(
session.close()
async def _replace_content_chunks(
session: Session, *, doc: Document, llm: Embedder, content: str, rel: str
) -> int:
"""Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace
the doc's content chunk rows (embeddings NULL), and embed the new
chunks in batches — returning the chunk count. The caller owns the
transaction (it commits, or rolls back on the :class:`EmbeddingError`
that propagates when the token-cap retry gives up).
The endpoint token-cap retry (halve this file's chunk target, floor
400 — URL-dense paragraphs tokenize at ~1.1 chars/token; the global
policy stays intact for the rest of the KB) is the house rule, shared
verbatim by the text path (:func:`_index_file`), the image path
(task 03), and the task-04 enrichment pass — one copy, three call
sites.
"""
settings = llm.settings
target = max(400, settings.chunk_target_chars)
chunks_text: list[str] = []
while True:
chunks_text = chunk_document(content, rel, target, settings.chunk_overlap_chars)
doc.chunks = [
Chunk(document_id=doc.id, position=i, content=c) for i, c in enumerate(chunks_text)
]
session.flush() # delete-orphan cascade drops the previous rows
if not doc.chunks:
break
try:
vectors = await llm.embed([c.content for c in doc.chunks])
for row, vec in zip(doc.chunks, vectors, strict=True):
row.embedding = vec
break
except EmbeddingError as e:
if "token cap" not in str(e) or target <= 400:
raise
logger.info(
"import: re-chunking at %d chars after endpoint token cap: %s",
target // 2,
rel,
)
target //= 2
return len(chunks_text)
async def _index_file(
session: Session,
*,
@@ -627,42 +734,34 @@ async def _index_file(
session.flush() # guarantees doc.id even for brand-new rows
# Phase 125 (task 04, LOCKED A1): a markdown file's image-reference
# bookkeeping — references whose image has NO document yet in this
# source are recorded on the doc's ``image_context`` (the
# ``IMAGE_CONTEXT:`` record) so the post-index enrichment pass picks
# them up when the image is later indexed; references that DO name an
# existing image doc record nothing (the task-03 path owns that case
# — the field stays NULL). The record rides the file's own
# transaction (committed by the ``session.commit()`` below).
if full_path.suffix.lower() in _MARKDOWN_EXTENSIONS:
doc.image_context = _missing_image_context(
session, source=source, md_rel=rel, content=content
)
# Phase 1+2 — chunk, replace the chunk rows (embeddings NULL), embed,
# and commit the whole file atomically. If the endpoint rejects a chunk
# as over its input token cap (URL-dense paragraphs tokenize at ~1.1
# chars/token), halve this file's chunk target and retry — the global
# policy stays intact for the rest of the KB.
target = max(400, settings.chunk_target_chars)
while True:
chunks_text = chunk_document(content, rel, target, settings.chunk_overlap_chars)
doc.chunks = [
Chunk(document_id=doc.id, position=i, content=c) for i, c in enumerate(chunks_text)
]
session.flush() # delete-orphan cascade drops the previous rows
if not doc.chunks:
break
try:
vectors = await llm.embed([c.content for c in doc.chunks])
for row, vec in zip(doc.chunks, vectors, strict=True):
row.embedding = vec
break
except EmbeddingError as e:
if "token cap" not in str(e) or target <= 400:
raise
logger.info(
"import: re-chunking at %d chars after endpoint token cap: %s",
target // 2,
rel,
)
target //= 2
# and commit the whole file atomically (the endpoint token-cap retry
# lives in the shared helper — the global policy stays intact for the
# rest of the KB).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
)
session.commit()
if verb == "added":
summary.added += 1
else:
summary.updated += 1
summary.chunks += len(chunks_text)
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, len(chunks_text))
summary.chunks += chunks_n
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
# Phase 30; phase 118 (A2, 2026-09-15): EVERY new/changed document
# gets a ``lite``-model summary — markdown included. Phase 30's
@@ -756,8 +855,409 @@ async def _store_summary(
logger.error("import: summary failed source=%s path=%s — %s", source, rel, e)
def _source_root_for(full_path: Path, rel: str) -> Path:
"""The source root of one walked file (phase 125, task 03).
The walk's invariant is ``full_path == root / rel`` (``rel`` is the
source-relative POSIX path, ``full_path`` the absolute path), so the
root is *full_path*'s parent walked UP by the number of DIRECTORY
components of *rel* (a top-level file has none — its own directory
IS the root).
"""
root = full_path.parent
for _ in PurePosixPath(rel).parts[:-1]:
root = root.parent
return root
def _resolve_ref_path(ref_path: str, md_rel: str) -> str | None:
"""One ``![alt](path)`` reference path → the image's source-relative
POSIX path, or ``None`` when the reference names no in-source file
(phase 125 — the single resolution rule shared by the task-03
context scan, the task-04 markdown bookkeeping, and the task-04
post-index enrichment pass).
A leading ``/`` = the source root (a root-absolute doc-style
reference); otherwise the path is resolved against the markdown
file's directory (``md_rel`` is the markdown's source-relative
POSIX path — its ``dirname``, so a top-level markdown joins against
the root). The result is normalized; a normalized path that still
escapes (``..``-prefixed — e.g. ``../../other/x.png``) or names no
file (``.``) matches no image document in this source, ever.
"""
if ref_path.startswith("/"):
resolved = ref_path.lstrip("/")
else:
resolved = posixpath.join(posixpath.dirname(md_rel), ref_path)
normalized = posixpath.normpath(resolved)
if normalized in ("", ".") or normalized.startswith(".."):
return None
return normalized
def _ref_points_at(ref: ImageRef, md_rel: str, image_rel: str) -> bool:
"""Does *ref* point at the image at *image_rel* (phase 125, task 03)?
The reference path is resolved against the markdown file's directory
(``md_rel`` is the markdown's source-relative POSIX path) — or,
when it starts with ``/``, against the SOURCE ROOT (a root-absolute
doc-style reference) — then normalized and compared for equality
with the image's source-relative path. Handles both the bare
relative reference (``pic.png`` next to the markdown) and the
folder-carrying one (``folder/pic.png`` from the root); a reference
to a different file that shares the filename is NOT a match.
"""
resolved = _resolve_ref_path(ref.path, md_rel)
return resolved is not None and resolved == image_rel
def _markdown_image_context(
root: Path,
llm: Embedder,
*,
source: str,
image_rel: str,
) -> str | None:
"""The combined markdown reference context for one image (phase 125,
task 03, LOCKED A1/A2) — or ``None`` when no markdown file in the
source references it (the standalone-image case: the whole phase-125
machinery is a no-op, byte-identical to pre-phase).
Scans the source's importable MARKDOWN files (``.md``/``.markdown``
under the source's import-extension set — hidden/excluded-dir rules
and the extension filter apply exactly as to the walk) and keeps
every :func:`~app.rag.md_images.extract_image_refs` reference that
POINTS AT this image (:func:`_md_ref_block` resolves each reference
via :func:`_resolve_ref_path` — ``pic.png`` from ``guide.md`` →
``pic.png``; ``../pic.png`` from ``sub/guide.md`` → ``pic.png``;
``folder/pic.png`` from ``guide.md`` → ``folder/pic.png``), one
block per referencing markdown file in the walk's sorted order.
The result is both the *context* argument of the vision call (the
model learns WHAT the image is for) and the stored
``image_context`` (the retrieval display's context line, task 05).
"""
blocks: list[str] = []
for md_path in iter_importable_files(root, llm.settings.import_extension_set):
if md_path.suffix.lower() not in _MARKDOWN_EXTENSIONS:
continue
try:
content = md_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
except OSError:
continue # unreadable source file — no context from it
block = _md_ref_block(source, md_path.relative_to(root).as_posix(), content, image_rel)
if block is not None:
blocks.append(block)
return "\n\n".join(blocks) if blocks else None
def _md_ref_block(source: str, md_rel: str, content: str, image_rel: str) -> str | None:
"""One ``Referenced in <source>/<md>:`` block for one markdown's
content (phase 125) — the shared builder of the task-03 disk scan
(:func:`_markdown_image_context`) and the task-04 pass
(:func:`_pending_image_context`).
Keeps every image reference in *content* that POINTS AT *image_rel*
(the resolution rule: :func:`_resolve_ref_path` — a reference to a
different file that merely shares the filename never matches,
LOCKED A1), in line order, rendered as::
Referenced in <source>/<markdown_path>:
- Alt text: "<alt_text>"
Context: <context_paragraphs>
(the ``Context:`` line is omitted for a reference with no
surrounding paragraphs). ``None`` when *content* has no reference
to this image.
"""
refs = [ref for ref in extract_image_refs(content) if _ref_points_at(ref, md_rel, image_rel)]
if not refs:
return None
lines = [f"Referenced in {source}/{md_rel}:"]
for ref in refs:
lines.append(f'- Alt text: "{ref.alt_text}"')
if ref.context:
lines.append(f" Context: {ref.context}")
return "\n".join(lines)
#: Header line of a markdown doc's pending-image record (phase 125,
#: task 04, LOCKED A1) — the stored ``image_context`` of a markdown
#: file that references images with no document yet in the source.
#: :func:`_parse_missing_image_refs` is its parse target (the record is
#: the post-index enrichment pass's reference index; the header is
#: ignored by the parser — the bullets are the data).
IMAGE_CONTEXT_HEADER = "IMAGE_CONTEXT:"
def _format_missing_image_refs(refs: Sequence[ImageRef], source: str) -> str:
"""One markdown's missing-image references → the ``IMAGE_CONTEXT:``
record stored on the markdown doc's ``image_context`` (phase 125,
task 04, LOCKED A1) — the durable record that "this markdown
referenced these images, which have no document yet in this
source":
.. code-block:: none
IMAGE_CONTEXT:
- [image.png] (line 42): "Alt text here" — 2 paragraphs of context
- [diagram.jpg] (line 100): "" — 1 paragraph of context
One bullet per reference, in reference order: the RESOLVED
source-relative image path in the brackets (the caller resolves via
:func:`_resolve_ref_path` — the post-index pass matches it against
the image doc's ``path``, so the resolution must agree with the
task-03 scan's), the 1-based line number, the alt text between the
quotes (``""`` for ``![](path)``), and the count of the reference's
context paragraphs (the task-02 window — the paragraphs themselves
live on the markdown doc's ``content``; the pass re-extracts them:
the record is the index, the content is the source). The shape is
exactly what :func:`_parse_missing_image_refs` parses back (the
"task 04b" parse target). *source* stays on the signature so the
call site reads like the record's home (the record rides a doc row
whose source is already known — display context, not data).
Empty *refs* → ``""`` (the caller stores ``None`` for "no record"
— an empty record would be a non-NULL empty string, and the pass's
``IS NOT NULL`` gate must see "no record" as NULL, not ``""``).
"""
if not refs:
return ""
lines = [IMAGE_CONTEXT_HEADER]
for ref in refs:
n = len(ref.context.split("\n\n")) if ref.context else 0
unit = "paragraph" if n == 1 else "paragraphs"
lines.append(
f'- [{ref.path}] (line {ref.line_number}): "{ref.alt_text}"'
f" — {n} {unit} of context"
)
return "\n".join(lines)
#: One bullet of the ``IMAGE_CONTEXT:`` record — the exact output shape
#: of :func:`_format_missing_image_refs`: the resolved image path in
#: the brackets, the 1-based line number, the alt text between the
#: quotes, the context-paragraph count.
_MISSING_REF_LINE_RE = re.compile(
r'^- \[(?P<path>[^\]]+)\] \(line (?P<line>\d+)\): "(?P<alt>.*)" — '
r"(?P<count>\d+) paragraph(?:s)? of context$"
)
def _parse_missing_image_refs(text: str) -> list[tuple[str, int, str]]:
"""Parse a markdown doc's ``image_context`` record back into
``(image path, line number, alt text)`` triples (phase 125, task 04
— the enrichment pass's reference index).
Only bullets in :func:`_format_missing_image_refs`' exact shape
parse (the greedy alt field still pins an alt that itself contains
``"``, as long as it does not contain the ``" — N paragraph…"``
tail): the header line, an owner-edited value, or a future shape
yields no references and the pass simply finds no match (fail-soft
— a malformed record must never raise into the sync).
"""
refs: list[tuple[str, int, str]] = []
for line in text.splitlines():
m = _MISSING_REF_LINE_RE.match(line)
if m is not None:
refs.append((m.group("path"), int(m.group("line")), m.group("alt")))
return refs
def _missing_image_context(
session: Session, *, source: str, md_rel: str, content: str
) -> str | None:
"""The markdown doc's ``image_context`` for one (re)indexed markdown
file (phase 125, task 04, LOCKED A1): the ``IMAGE_CONTEXT:`` record
of every ``![alt](path)`` reference whose image has NO ``is_image``
document yet in this source, or ``None`` when every in-source
reference names an existing image doc (the task-03 path owns those
— enrichment at image-index time — and the field stays NULL) or the
file has no references at all.
Each reference is resolved exactly as the task-03 scan
(:func:`_resolve_ref_path`); a reference that escapes the source
(e.g. ``../../other/x.png``) names no in-source image document, ever,
and is not recorded (the field is the pending-enrichment index, not
a general reference dump). A re-indexed markdown recomputes the
field from its CURRENT content — the image now exists (or the
reference was dropped) → the field drops back to ``None``.
"""
refs = extract_image_refs(content)
if not refs:
return None
existing = set(
session.scalars(
select(Document.path).where(
Document.source == source, Document.is_image.is_(True)
)
).all()
)
missing: list[ImageRef] = []
for ref in refs:
resolved = _resolve_ref_path(ref.path, md_rel)
if resolved is None or resolved in existing:
continue
missing.append(
ImageRef(
path=resolved,
alt_text=ref.alt_text,
context=ref.context,
line_number=ref.line_number,
)
)
if not missing:
return None
return _format_missing_image_refs(missing, source)
def _pending_image_context(
source: str, image_rel: str, md_docs: Sequence[Document]
) -> str | None:
"""The task-04 pass's context for one pending image doc: the SAME
combined string the task-03 scan builds (``Referenced in …`` blocks
joined with a blank line) — but sourced from the markdown DOC ROWS
instead of the on-disk files: a markdown doc contributes only when
its stored ``image_context`` record lists the image's source-relative
path (the durable "this markdown referenced this image" record,
work 1), and the reference (alt text + surrounding paragraphs) is
re-extracted from the doc's stored ``content`` (the full text is on
the row — the record is the index, the content is the source).
``None`` when no markdown doc references the image.
"""
blocks: list[str] = []
for md in md_docs:
listed = _parse_missing_image_refs(md.image_context or "")
if image_rel not in {path for path, _, _ in listed}:
continue
block = _md_ref_block(source, md.path, md.content, image_rel)
if block is not None:
blocks.append(block)
return "\n\n".join(blocks) if blocks else None
async def _enrich_pending_images(
session: Session, *, source: str, llm: Embedder, summary: ImportSummary
) -> None:
"""The phase-125 task-04 post-index enrichment pass — once per
source, after ALL of its files are indexed (:func:`import_sources`).
Closes the markdown-FIRST case: a markdown file indexed while its
referenced image had no document yet stores the reference on its
``image_context`` (work 1 — the ``IMAGE_CONTEXT:`` record). When the
image is later indexed but the task-03 disk scan could not see the
reference (the markdown file no longer on disk — a deleted / moved
source file whose doc row survives a ``prune=False`` run — or any
other disk-vs-DB drift), the image doc is still ``image_context IS
NULL`` and this pass enriches it from the markdown doc's STORED rows
(see :func:`_pending_image_context`).
The enrichment is the task-03 mechanics on the existing doc: a fresh
context-aware description (``describe_image`` with the context — the
description remains the primary content, LOCKED A3), ``content`` =
``<context>\\n\\n<description>`` (and so the ``summary`` — the
phase-122 ``summary = content`` convention), the context stored on
``image_context``, and the re-chunked + re-embedded pipeline (the
re-describe reads the image's PERSISTENT copy — the source file may
be the one that is gone).
Idempotent by contract (ASSUMPTION 4): an image doc with
``image_context`` SET is never touched (already enriched — the
task-03 path or an earlier pass), and a markdown doc's
``image_context`` is NOT cleared on success (it stays the retrieval
display's record, task 05). Fail-soft: a missing persistent copy, a
failed/empty re-description, or an embed error keeps the image doc
as-is (warned / counted) and the next sync retries it; one bad image
never blocks the others. A success counts ``updated`` + ``chunks``
(the doc's content really changed — the caller's KB-change gate and
the sources-version bump both fire on ``added + updated``, which is
correct here) and a dedicated ``image-context-enriched`` log line
(PLAN §9).
"""
pending = session.scalars(
select(Document).where(
Document.source == source,
Document.is_image.is_(True),
Document.image_context.is_(None),
)
).all()
if not pending:
return
md_docs = session.scalars(
select(Document).where(
Document.source == source,
Document.is_image.is_(False),
Document.image_context.is_not(None),
)
).all()
if not md_docs:
return
for image_doc in pending:
context = _pending_image_context(source, image_doc.path, md_docs)
if context is None:
continue # no markdown doc records a reference to this image
if not image_doc.image_path or not Path(image_doc.image_path).is_file():
# No persistent bytes to re-describe — keep the doc as-is
# (the next sync retries; a copy gone for good leaves an
# unenrichable doc, and the warning is the signal).
summary.images_failed += 1
logger.warning(
"import: image-context enrichment skipped (no persistent copy) "
"source=%s path=%s",
source, image_doc.path,
)
continue
copy_path = Path(image_doc.image_path)
description = await _describe_or_skip(
llm,
data=copy_path.read_bytes(),
source=source,
rel=image_doc.path,
full_path=copy_path,
context=context,
)
if description is None:
summary.images_failed += 1
logger.warning(
"import: image re-description failed source=%s path=%s",
source, image_doc.path,
)
continue
try:
image_doc.image_context = context
image_doc.content = f"{context}\n\n{description}"
chunks_n = await _replace_content_chunks(
session, doc=image_doc, llm=llm,
content=image_doc.content, rel=image_doc.path,
)
session.commit()
except EmbeddingError as e:
session.rollback()
summary.errors += 1
logger.error(
"import: image-context enrichment failed source=%s path=%s — %s",
source, image_doc.path, e,
)
continue
summary.updated += 1
summary.chunks += chunks_n
logger.info(
"import: image-context-enriched source=%s path=%s chunks=%d",
source, image_doc.path, chunks_n,
)
await _store_summary(
session, doc=image_doc, source=source, rel=image_doc.path,
content=image_doc.content, llm=llm, summary=summary,
)
async def _describe_or_skip(
llm: Embedder, *, data: bytes, source: str, rel: str, full_path: Path
llm: Embedder,
*,
data: bytes,
source: str,
rel: str,
full_path: Path,
context: str | None = None,
) -> str | None:
"""Phase 122 — the SINGLE seam for the image description (task 03).
@@ -777,9 +1277,17 @@ async def _describe_or_skip(
description error). ``source``/``rel`` stay on the signature so the
caller (and the patch) reads like the document being described;
the failure's doc identity is logged by the caller's warning.
``context`` (phase 125, task 03) is the optional markdown reference
context (:func:`_markdown_image_context`), forwarded verbatim to
:func:`app.rag.summarizer.describe_image` — appended AFTER the
locked describe prompt so the vision model can write a
context-aware description (LOCKED A3: the description remains the
primary content). ``None`` (standalone image) keeps the message
byte-identical to pre-phase-125.
"""
mime = IMAGE_MIMES.get(full_path.suffix.lower(), IMAGE_FALLBACK_MIME)
return await describe_image(llm, data=data, mime=mime)
return await describe_image(llm, data=data, mime=mime, context=context)
def _delete_image_copy(image_path: str | None) -> None:
@@ -833,6 +1341,20 @@ async def _index_image_file(
Fail-soft (LOCKED A3): a failed/empty description SKIPS the doc
entirely (no row, no copy) — ``summary.images_failed`` + a warning,
the sync continues.
Phase 125 (task 03, LOCKED A1/A2/A3): a NEW/CHANGED image first
scans the same source's markdown files for references to it (see
:func:`_markdown_image_context`). When one or more exist, the
combined context string goes to the vision call (``context`` — the
model writes a context-aware description; the description remains
the primary content) and is stored on the doc's ``image_context``;
the doc's ``content`` becomes ``<context>\n\n<description>`` (and so
does ``summary`` — the phase-122 ``summary = content`` convention —
as do the embedded chunks). No reference → ``image_context`` stays
NULL and ``content`` is the bare description (byte-identical to
pre-phase). The UNCHANGED path never re-scans (the stored context
is part of the row's committed content; a changed source state
re-indexes the file through this path).
"""
settings = llm.settings
data = full_path.read_bytes()
@@ -880,12 +1402,24 @@ async def _index_image_file(
return
verb = "updated" if doc is not None else "added"
# Phase 125 (task 03, LOCKED A1/A2): BEFORE the description — the
# markdown reference context for this image (the alt text +
# surrounding paragraphs of the ``![alt](path)`` references that
# name it in the same source), or ``None`` (standalone image — the
# whole phase-125 enrichment below is a no-op, byte-identical to
# pre-phase). It goes to the vision model (a context-aware
# description, LOCKED A3) AND is stored on the doc (``image_context``
# — the retrieval display's context line, task 05).
image_context = _markdown_image_context(
_source_root_for(full_path, rel), llm, source=source, image_rel=rel
)
# The description is the doc's content (task 03: and its summary) —
# it is generated BEFORE anything is written, so a failure skips the
# doc with no row and no copy (the copy is only made after a
# successful description — a failure never leaves an orphan).
content = await _describe_or_skip(
llm, data=data, source=source, rel=rel, full_path=full_path
llm, data=data, source=source, rel=rel, full_path=full_path,
context=image_context,
)
if content is None:
# LOCKED A3 fail-soft: an undescribed image is unsearchable
@@ -893,6 +1427,13 @@ async def _index_image_file(
summary.images_failed += 1
logger.warning("import: image description failed source=%s path=%s", source, rel)
return
if image_context is not None:
# Phase 125 (task 03): the enriched content — the reference
# context FIRST (for retrieval), then the description. The
# ``summary`` field mirrors this (the phase-122 image
# convention: ``summary = content``), and the chunk pipeline
# below embeds it.
content = f"{image_context}\n\n{content}"
# The persistent copy: uploads are replaced on every upload, git
# checkouts are re-cloned, local dirs are user-edited — the served
@@ -925,6 +1466,10 @@ async def _index_image_file(
created_at=normalize_doc_date(raw_date),
is_image=True,
image_path=str(copy_path),
# Phase 125 (task 03, LOCKED A1): the referencing markdown's
# alt text + surrounding paragraphs (``None`` for a
# standalone image — the pre-phase row shape).
image_context=image_context,
)
session.add(doc)
else:
@@ -940,45 +1485,28 @@ async def _index_image_file(
doc.created_at_manual = False
doc.is_image = True
doc.image_path = str(copy_path)
# Phase 125 (task 03): a re-indexed image re-scans its markdown
# references — the context follows the current source state
# (a deleted referencing markdown drops it back to ``None``).
doc.image_context = image_context
session.flush() # guarantees doc.id even for brand-new rows
# Phase 1+2 — the UNCHANGED pipeline on the description: chunk,
# replace the chunk rows (embeddings NULL), embed, and commit the
# whole file atomically (one transaction per file). The token-cap
# retry loop is copied from the text path; a description is short,
# so it never fires in practice.
target = max(400, settings.chunk_target_chars)
while True:
chunks_text = chunk_document(content, rel, target, settings.chunk_overlap_chars)
doc.chunks = [
Chunk(document_id=doc.id, position=i, content=c) for i, c in enumerate(chunks_text)
]
session.flush() # delete-orphan cascade drops the previous rows
if not doc.chunks:
break
try:
vectors = await llm.embed([c.content for c in doc.chunks])
for row, vec in zip(doc.chunks, vectors, strict=True):
row.embedding = vec
break
except EmbeddingError as e:
if "token cap" not in str(e) or target <= 400:
raise
logger.info(
"import: re-chunking at %d chars after endpoint token cap: %s",
target // 2,
rel,
)
target //= 2
# Phase 1+2 — the same pipeline on the description (chunk, replace
# the chunk rows (embeddings NULL), embed, commit — one transaction
# per file; the endpoint token-cap retry lives in the shared helper,
# and a description is short, so it never fires in practice).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
)
session.commit()
if verb == "added":
summary.added += 1
else:
summary.updated += 1
summary.chunks += len(chunks_text)
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, len(chunks_text))
summary.chunks += chunks_n
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
# Phase 30 shape on the description — image-aware (task 03, LOCKED
# A3): the description IS the summary (stored verbatim, no
app/rag/md_images.py
+162
View File
@@ -0,0 +1,162 @@
"""Markdown image-reference extraction (phase 125, LOCKED A1).
Pure functions, no I/O — fully unit-testable. Stdlib only.
Markdown documents reference images inline (``![alt](path)``). The
paragraphs AROUND the reference carry the *why* — what the diagram
illustrates, which procedure it belongs to — context that the
standalone vision description of the image itself (phase 122) does not
have. :func:`extract_image_refs` finds every image reference and
returns it with its alt text, its 1-based line number, and the
surrounding paragraph context: up to *n* paragraphs before and after
the reference paragraph (default :data:`CONTEXT_PARAGRAPHS`).
Paragraph semantics mirror the chunker (PLAN §5): paragraphs are
separated by blank lines and code fences are **atomic** — a fenced
block (possibly containing blank lines) is ONE paragraph, so it counts
as a single unit of the context window. Only the *references* inside a
fence are skipped: there they are literal text, not actual images.
"""
from __future__ import annotations
import re
from collections.abc import Sequence
from dataclasses import dataclass
#: Opening/closing code fence (three or more backticks or tildes,
#: optionally indented) — the SAME semantics as the chunker's fence
#: handling (PLAN §5), so "fence" means the same thing here as in
#: chunking.
_FENCE_RE = re.compile(r"^(`{3,}|~{3,})")
#: Markdown image reference ``![alt](path)``. The alt may be EMPTY
#: (``![](path)``); the path may contain spaces but not ``)`` — nested
#: parentheses inside a path are accepted as a miss (rare in practice,
#: phase 125 task 02).
_IMAGE_REF_RE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
#: Default number of context paragraphs BEFORE and AFTER the reference
#: paragraph (LOCKED A1: "alt text + N paragraphs before/after the
#: reference") — 2 gives ~2-4 sentences of surrounding text.
CONTEXT_PARAGRAPHS: int = 2
@dataclass
class ImageRef:
"""One extracted ``![alt](path)`` reference with its context window.
* ``path`` — the image path (relative or absolute; surrounding
whitespace stripped);
* ``alt_text`` — the alt text (may be empty for ``![](path)``);
* ``context`` — the surrounding paragraphs: up to *n* before and
*n* after the reference paragraph, joined with blank lines. The
reference paragraph itself is NOT included (the reference is
already captured as ``alt_text`` + ``path``);
* ``line_number`` — 1-based line number of the reference.
"""
path: str
alt_text: str
context: str
line_number: int
def _fence_flags(lines: Sequence[str]) -> list[bool]:
"""Per-line flags: ``True`` on a fence delimiter or inside a fence.
Mirrors the chunker (PLAN §5): an opener is three or more backticks
or tildes (indented allowed); while open, EVERY line — including
blank ones — is in-fence until the first line starting with the
same three-character marker.
"""
flags: list[bool] = []
marker: str | None = None
for line in lines:
stripped = line.lstrip()
if marker is None:
m = _FENCE_RE.match(stripped)
if m:
marker = m.group(1)[:3]
flags.append(True)
else:
flags.append(False)
else:
flags.append(True)
if stripped.startswith(marker):
marker = None
return flags
def _paragraph_blocks(
lines: Sequence[str], flags: Sequence[bool]
) -> tuple[list[tuple[int, int]], list[int]]:
"""Paragraph blocks as half-open (start, end) line ranges, plus a
per-line block-index map (``-1`` on blank outside-fence lines).
Blocks are separated by blank lines *outside* fences; a fenced code
block (possibly containing blank lines) is ONE block — atomic (the
same paragraph semantics the chunker chunks with, PLAN §5).
"""
blocks: list[tuple[int, int]] = []
start: int | None = None
for i, (line, in_fence) in enumerate(zip(lines, flags, strict=True)):
if in_fence or line.strip():
if start is None:
start = i
elif start is not None:
blocks.append((start, i))
start = None
if start is not None:
blocks.append((start, len(lines)))
line_block = [-1] * len(lines)
for bi, (s, e) in enumerate(blocks):
line_block[s:e] = [bi] * (e - s)
return blocks, line_block
def extract_image_refs(
content: str, context_paragraphs: int = CONTEXT_PARAGRAPHS
) -> list[ImageRef]:
"""Extract every markdown image reference in *content*, with context.
Each ``![alt](path)`` reference OUTSIDE a code fence becomes one
:class:`ImageRef` (multiple references on one line are extracted
separately, each with the same context window). ``context`` joins
the up-to-*context_paragraphs* paragraphs BEFORE and AFTER the
reference paragraph with blank lines; the reference paragraph
itself is excluded (the reference is already captured as
``alt_text`` + ``path``), and a reference at the start/end of the
file gets whatever paragraphs are available. Fences are atomic
paragraphs (one fence block = one context unit), and references
inside one are literal text — skipped.
A negative *context_paragraphs* raises :class:`ValueError` (fail
loud); 0 means "no surrounding context".
"""
if context_paragraphs < 0:
raise ValueError("context_paragraphs must be >= 0")
lines = content.splitlines()
if not lines:
return []
flags = _fence_flags(lines)
blocks, line_block = _paragraph_blocks(lines, flags)
refs: list[ImageRef] = []
for lineno, (line, in_fence) in enumerate(zip(lines, flags, strict=True), start=1):
if in_fence:
continue
for m in _IMAGE_REF_RE.finditer(line):
# The reference line is non-blank, outside-fence content, so
# it always belongs to a paragraph block (>= 0).
bi = line_block[lineno - 1]
before = blocks[max(0, bi - context_paragraphs) : bi]
after = blocks[bi + 1 : bi + 1 + context_paragraphs]
context = "\n\n".join("\n".join(lines[s:e]) for s, e in before + after)
refs.append(
ImageRef(
path=m.group(2).strip(),
alt_text=m.group(1).strip(),
context=context,
line_number=lineno,
)
)
return refs
app/rag/prompts.py
+137 -7
View File
@@ -89,6 +89,7 @@ no longer injected — see :mod:`app.api.chat`).
"""
from __future__ import annotations
import re
from collections.abc import Sequence
from typing import Any
@@ -434,6 +435,116 @@ def build_kb_section(overview: str, max_chars: int | None = None) -> str:
return ""
#: Phase 125 (task 05, LOCKED A4) — the header line of one block of an
#: enriched image doc's stored ``image_context`` (the exact shape the
#: importer's reference-block writer emits, tasks 03/04):
#: ``Referenced in <source>/<markdown_path>:`` — a block is one
#: referring markdown file, and a block's line FOLLOWING its header is
#: always its first ``- Alt text: "..."`` bullet. A look-alike line in
#: the context body without that bullet is spurious and dropped
#: (graceful degradation, A4).
_IMAGE_CTX_HEADER_RE = re.compile(r"^Referenced in (?P<ref>.+):$")
#: The ``- Alt text: "..."`` bullet of the same block. The greedy alt
#: field (the block writer does not escape quotes) is captured between
#: the opening quote (the pattern's literal) and the CLOSING quote
#: (the pattern's trailing ``"$``) — an alt with inner ``"`` survives
#: intact (the greedy walk keeps the inner quote, the final one is the
#: delimiter; an empty alt matches as the empty group).
_IMAGE_CTX_ALT_RE = re.compile(r'^- Alt text: "(?P<alt>.*)"$')
def _image_ctx_clause(ref: str, alt: str | None) -> str:
"""One clause of the retrieval context line (phase 125, task 05):
the referring markdown's stored identity (``<source>/<path>``
exactly as the block header carries it) plus the reference's alt
text quoted after a colon — the reference's purpose in one phrase.
A reference with no alt (``![](path)``) renders the bare identity
(nothing to quote).
"""
if alt:
return f'{ref}: "{alt}"'
return ref
def _extract_context_line(image_context: str) -> str | None:
"""The ONE-LINE rendering of an enriched image doc's stored
``image_context`` (phase 125, task 05, LOCKED A4) — or ``None``
when nothing in the value parses as a reference block (graceful
degradation: the ``<document>`` block then shows only the
description, exactly pre-phase — the value never raises into the
prompt build).
The stored value (tasks 03/04) is one block per referring markdown
file, blank-line joined::
Referenced in <source>/<markdown_path>:
- Alt text: "<alt_text>"
Context: <context paragraphs — may span many lines>
- Alt text: "<a second reference's alt>"
The result keeps only what tells the LLM WHY the image exists in
the documentation — one clause per referring file (its identity +
the FIRST reference's alt), after a single ``Referenced in ``
prefix, ``; ``-joined::
Referenced in docs/a.md: "alt a"; docs/b.md
The ``Context:`` body is never rendered (the surrounding paragraphs
already reached the vision model at enrichment time, LOCKED A3 —
the line is the locator + purpose, not a duplicate of the
paragraphs), so the line stays short enough not to bloat the
prompt. The markdown doc's ``IMAGE_CONTEXT:`` pending record
(task 04) carries no ``Referenced in`` header at all and parses to
``None`` — and it rides a TEXT doc, which :func:`_document_body`
excludes from the line via the ``is_image`` guard anyway.
"""
clauses: list[str] = []
ref: str | None = None # the current block's "source/markdown_path"
alt: str | None = None # the current block's first bullet's alt text
expecting_alt = False # the line after a header must be its bullet
for line in image_context.splitlines():
header = _IMAGE_CTX_HEADER_RE.match(line)
if header is not None:
if ref is not None and not expecting_alt:
clauses.append(_image_ctx_clause(ref, alt))
ref, alt = header.group("ref"), None
expecting_alt = True
continue
alt_match = _IMAGE_CTX_ALT_RE.match(line) if ref is not None else None
if expecting_alt:
if alt_match is None:
# Spurious header (a look-alike line in context text):
# its bullet never came — drop the block.
ref, alt, expecting_alt = None, None, False
else:
alt = alt_match.group("alt")
expecting_alt = False
continue
# Any other line is context body or a second bullet of the
# same block — the first reference's alt is already captured,
# so the one-line rendering is unaffected (no branch needed:
# with a live block and no pending bullet, ``alt`` is never
# ``None``).
if ref is not None and not expecting_alt:
clauses.append(_image_ctx_clause(ref, alt))
if not clauses:
return None
return "Referenced in " + "; ".join(clauses)
def _image_context_line(doc: Document) -> str | None:
"""The phase-125 context line for one suggested document (task 05,
LOCKED A4) — ``None`` in the common case (EVERY text doc, and every
image doc without a parsable ``image_context`` — standalone /
pre-phase rows: the byte-identical path) and the one-line
:func:`_extract_context_line` rendering otherwise (``is_image``
docs with a non-NULL, parsable ``image_context`` only)."""
if not doc.is_image or not doc.image_context:
return None
return _extract_context_line(doc.image_context)
def _document_body(doc: Document) -> str:
"""The body of one ``<document>`` suggestion block (phase 118, LOCKED
A6): the document's stored summary (stripped) — NEVER the full
@@ -447,15 +558,33 @@ def _document_body(doc: Document) -> str:
summary-bearing docs makes no settings call for the cap). Content at
or under the cap rides whole, unmarked (nothing was cut). No LLM
call at chat time — the preview is deterministic.
Phase 125 (task 05, LOCKED A4): an ENRICHED image doc (``is_image``
with a non-NULL ``image_context`` that parses — see
:func:`_image_context_line`) leads its body with the ONE short
context line — ``Referenced in <source>/<path>: "<alt>"`` — before
the description, so the LLM knows WHY the image exists in the
documentation, not only what it shows. Every other doc is
byte-identical to the pre-phase body: every text doc (its
``image_context`` may carry the task-04 pending record, which never
renders) and every non-enriched image doc (NULL ``image_context``
— the phase's byte-identity criterion).
"""
summary = (doc.summary or "").strip()
if summary:
return summary
limit = get_settings().suggestion_preview_chars
content = doc.content
if len(content) > limit:
return content[:limit] + "\n" + TRUNCATION_MARKER
return content
body = summary
else:
limit = get_settings().suggestion_preview_chars
content = doc.content
body = (
content[:limit] + "\n" + TRUNCATION_MARKER
if len(content) > limit
else content
)
context_line = _image_context_line(doc)
if context_line is not None:
body = context_line + "\n\n" + body
return body
def build_high_prompt(
@@ -530,7 +659,8 @@ def build_high_prompt(
APPENDED after ``title`` — the only position; always present,
``created_at`` is NOT NULL) — plus the document's summary
(:func:`_document_body`; the NULL/whitespace-summary preview
fallback is LOCKED A5).
fallback is LOCKED A5; an enriched image doc's summary leads with
the phase-125 one-line context line, LOCKED A4).
"""
# Phase 106 (D5): every document the model sees carries its
# creation date — the block's ``date`` attribute (the row's
app/rag/summarizer.py
+23 -6
View File
@@ -211,6 +211,7 @@ async def describe_image(
data: bytes,
mime: str,
settings: Settings | None = None,
context: str | None = None,
) -> str | None:
"""One-shot CHAT-model (vision) description of one image (phase 122,
LOCKED A3) — the text that becomes the image document's ``content``
@@ -220,11 +221,22 @@ async def describe_image(
ONE chat-model call against ``settings.llm_chat_model`` (the vision
model — the ``lite`` summary model is NOT assumed vision-capable)
with the multimodal user message the OpenAI-compatible API expects:
``[{type: "text", text: DESCRIBE_PROMPT}, {type: "image_url",
image_url: {url: <data URL from *data* + *mime*>}}]`` — no system
prompt, no tools, no app-level retries beyond the client's own
(SDK-level + the house one-shot empty-content policy) — a
description failure must not stall a sync.
``[{type: "text", text: DESCRIBE_PROMPT + context prefix},
{type: "image_url", image_url: {url: <data URL from *data* +
*mime*>}}]`` — no system prompt, no tools, no app-level retries
beyond the client's own (SDK-level + the house one-shot empty-
content policy) — a description failure must not stall a sync.
*context* (phase 125, task 03, LOCKED A3): the optional markdown
reference context — the alt text + surrounding paragraphs of the
``![alt](path)`` references that point at THIS image (the importer's
same-source scan). When provided, it is appended to the text part
AFTER the locked :data:`DESCRIBE_PROMPT` (the marker + instruction
stay byte-identical — the ``IMAGE_DESCRIPTION_MODE`` mock key is
untouched): the vision model is told WHAT THE IMAGE IS FOR so it
can write a context-aware description. ``context=None`` (the
default — standalone images) sends the bare prompt: the message is
byte-identical to the pre-phase-125 shape.
Returns the stripped reply cut exactly at
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap —
@@ -236,11 +248,16 @@ async def describe_image(
continues (LOCKED A3).
"""
data_url = image_data_url(data, mime)
context_prefix = (
f"\n\nThis image is referenced in documentation with the following context:\n{context}"
if context
else ""
)
messages: list[dict[str, Any]] = [
{
"role": "user",
"content": [
{"type": "text", "text": DESCRIBE_PROMPT},
{"type": "text", "text": DESCRIBE_PROMPT + context_prefix},
{"type": "image_url", "image_url": {"url": data_url}},
],
}