phase: 173_docs_push_creds_split
All gates green. Final report: **Phase 173 — final verification pass (all tasks already complete in `complete/`)** - Verified the full implementation is in the working tree: `docs_user`/`docs_token` settings + catalogue rows (task 01), `docs_repo` PUT 422 guard + `docs_token` write-only pin (task 02), `run_git(env)` + `push_document(user, token)` credential env (task 03), `settings_migrate.py` wired into `create_app()` (task 04), E2E suite (task 05). - **Completion criteria:** 1. GET `/api/admin/settings` lists `docs_user` (plain) + `docs_token` (secret, `value=null`), token never in any body — ✅ `test_docs_token_is_write_only_masked_end_to_end` + E2E sentinel-absent-from-DOM 2. PUT `docs_repo` with `://user:pass@` → 422 naming the fields; clean URL passes — ✅ integration + E2E UI-side 3. Token set → `GIT_CURL_USERNAME`/`GIT_CURL_PASSWORD`/`GIT_TERMINAL_PROMPT=0` on every git step; empty → `env=None` — ✅ mocked-git unit tests 4. Legacy env split once at startup, idempotent, one INFO line, `.env` untouched — ✅ `test_settings_migrate.py` (4 shapes) + `test_docs_push_migrate.py` 5. Full suite green, coverage >90%, lint clean — ✅ 6. E2E isolated: `test_docs_push_creds.py` 4/4; regression set (panel + layout + response_to_docs) 21/21 — ✅ 7. Commit + phase move — harness's job (per instructions, not mine). 8. README/AGENTS consistent — ✅ (below) - Tests/lint: `uv run pytest` exit 0 (4112 passed) · `uv run pytest --cov=app --cov-report=term-missing` TOTAL 99% · `uv run ruff check .` clean · `uv run pyright` 0 errors - Docs: README.md (incl. Docs-push config section, guard/migration paragraph, `.env.example` block) and AGENTS.md verified consistent in full; fixed two stale docstrings in `app/config.py` + `app/core/settings_catalog.py` (`is_set` → the API's actual `has_value` field). - No live-infrastructure changes (no live step in this phase). No next pending phase — `todo/` held only 173.
This commit is contained in:
+49
-10
@@ -22,6 +22,7 @@ stderr (the ``GitSyncError`` style), and the checkout under
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.git_sync import GitSyncError, run_git
|
||||
@@ -61,9 +62,28 @@ def push_document(
|
||||
rel_path: str,
|
||||
content: str,
|
||||
commit_message: str,
|
||||
user: str = "",
|
||||
token: str = "",
|
||||
) -> tuple[str, str]:
|
||||
"""Commit *content* at *rel_path* and push (ff-only) to *branch*.
|
||||
|
||||
*user* / *token* are the docs-push credentials (phase 173, C2):
|
||||
when *token* is non-blank, EVERY git step of the sequence runs with
|
||||
git's documented HTTP credential environment —
|
||||
``GIT_CURL_USERNAME`` / ``GIT_CURL_PASSWORD`` (+
|
||||
``GIT_TERMINAL_PROMPT=0`` so a headless server fails fast instead
|
||||
of hanging on a prompt) — merged over ``os.environ`` (the
|
||||
subprocess env-replacement trap: a dict REPLACES the parent
|
||||
environment, so the merge is what keeps ``PATH`` and friends).
|
||||
The credential travels only in that environment — it never appears
|
||||
in the argv (the clean *repo* URL stays in the argv untouched),
|
||||
so it is absent from the process list. ``GIT_CURL_*`` is inert for
|
||||
``ssh://`` / local-path remotes (accepted — the remote decides).
|
||||
**C5:** when *token* is blank (the default — an anonymous push to
|
||||
a public or credential-free remote, exactly the pre-phase state),
|
||||
every ``run_git`` call keeps ``env=None`` and the invocations are
|
||||
byte-identical to the pre-phase ones.
|
||||
|
||||
Step sequence (each step via :func:`scripts.git_sync.run_git`;
|
||||
``cwd=work_dir`` once the checkout exists):
|
||||
|
||||
@@ -108,6 +128,19 @@ def push_document(
|
||||
"""
|
||||
_refuse_unsafe_rel_path(rel_path)
|
||||
work = Path(work_dir).expanduser()
|
||||
# C5: blank token → env stays None → every invocation below is
|
||||
# byte-identical to the pre-phase one.
|
||||
git_env: dict[str, str] | None = None
|
||||
if token.strip():
|
||||
# C2: git's documented HTTP credential env — never in the argv.
|
||||
# GIT_TERMINAL_PROMPT=0 fails fast instead of hanging the headless
|
||||
# server on a credential prompt.
|
||||
git_env = {
|
||||
**os.environ,
|
||||
"GIT_CURL_USERNAME": user,
|
||||
"GIT_CURL_PASSWORD": token,
|
||||
"GIT_TERMINAL_PROMPT": "0",
|
||||
}
|
||||
try:
|
||||
if not (work / ".git").exists():
|
||||
work.parent.mkdir(parents=True, exist_ok=True)
|
||||
@@ -123,25 +156,30 @@ def push_document(
|
||||
str(work),
|
||||
],
|
||||
cwd=work.parent,
|
||||
env=git_env,
|
||||
)
|
||||
run_git(["git", "fetch", "--depth", "1", "origin", base_branch], cwd=work)
|
||||
run_git(["git", "fetch", "--depth", "1", "origin", base_branch], cwd=work, env=git_env)
|
||||
# Capture the fetched base tip now — a *failed* fetch clears
|
||||
# FETCH_HEAD, so the symbolic ref cannot be reused later.
|
||||
base_sha = run_git(["git", "rev-parse", "FETCH_HEAD"], cwd=work).strip()
|
||||
base_sha = run_git(["git", "rev-parse", "FETCH_HEAD"], cwd=work, env=git_env).strip()
|
||||
try:
|
||||
run_git(["git", "fetch", "--depth", "100", "origin", branch], cwd=work)
|
||||
branch_sha = run_git(["git", "rev-parse", "FETCH_HEAD"], cwd=work).strip()
|
||||
run_git(
|
||||
["git", "fetch", "--depth", "100", "origin", branch], cwd=work, env=git_env
|
||||
)
|
||||
branch_sha = run_git(
|
||||
["git", "rev-parse", "FETCH_HEAD"], cwd=work, env=git_env
|
||||
).strip()
|
||||
except GitSyncError:
|
||||
branch_sha = None # the branch does not exist on the remote yet
|
||||
if run_git(["git", "branch", "--list", branch], cwd=work).strip():
|
||||
run_git(["git", "checkout", branch], cwd=work)
|
||||
if run_git(["git", "branch", "--list", branch], cwd=work, env=git_env).strip():
|
||||
run_git(["git", "checkout", branch], cwd=work, env=git_env)
|
||||
else:
|
||||
anchor = branch_sha if branch_sha is not None else base_sha
|
||||
run_git(["git", "checkout", "-B", branch, anchor], cwd=work)
|
||||
run_git(["git", "checkout", "-B", branch, anchor], cwd=work, env=git_env)
|
||||
target = work / rel_path
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(content, encoding="utf-8")
|
||||
run_git(["git", "add", "--", rel_path], cwd=work)
|
||||
run_git(["git", "add", "--", rel_path], cwd=work, env=git_env)
|
||||
run_git(
|
||||
[
|
||||
"git",
|
||||
@@ -156,9 +194,10 @@ def push_document(
|
||||
commit_message,
|
||||
],
|
||||
cwd=work,
|
||||
env=git_env,
|
||||
)
|
||||
run_git(["git", "push", "origin", branch], cwd=work)
|
||||
sha = run_git(["git", "rev-parse", "HEAD"], cwd=work).strip()
|
||||
run_git(["git", "push", "origin", branch], cwd=work, env=git_env)
|
||||
sha = run_git(["git", "rev-parse", "HEAD"], cwd=work, env=git_env).strip()
|
||||
except GitSyncError as err:
|
||||
raise DocsPushError(str(err)) from err
|
||||
return branch, sha
|
||||
|
||||
@@ -11,8 +11,9 @@ to do with each of the ``Settings`` fields:
|
||||
presentation metadata (``group`` + ``label``) and its capability
|
||||
flags:
|
||||
|
||||
- ``secret`` — write-only (LOCKED A4): the API reports ``is_set``,
|
||||
never the plaintext; a saved value is masked on read.
|
||||
- ``secret`` — write-only (LOCKED A4): the API reports ``has_value``
|
||||
(and ``masked=True`` / ``value=None``), never the plaintext; a
|
||||
saved value is masked on read.
|
||||
- ``restart_required`` — boot-wired (LOCKED A4): the middleware /
|
||||
engine / logging / static mount read it once at boot, so the
|
||||
override applies on the NEXT boot (the panel shows the badge).
|
||||
@@ -301,6 +302,10 @@ CATALOG: dict[str, SettingMeta] = {
|
||||
"pdf_page_dir": SettingMeta("pdf_page_dir", "PDF", "PDF page dir"),
|
||||
# --- Docs push ---
|
||||
"docs_repo": SettingMeta("docs_repo", "Docs push", "Docs repo"),
|
||||
"docs_user": SettingMeta("docs_user", "Docs push", "Docs user"),
|
||||
"docs_token": SettingMeta(
|
||||
"docs_token", "Docs push", "Docs token", secret=True
|
||||
),
|
||||
"docs_branch": SettingMeta("docs_branch", "Docs push", "Docs branch"),
|
||||
"docs_base_branch": SettingMeta(
|
||||
"docs_base_branch", "Docs push", "Docs base branch"
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
"""One-time settings migrations (phase 173).
|
||||
|
||||
``migrate_legacy_docs_creds`` splits a legacy ``://user:pass@``
|
||||
``docs_repo`` value into ``docs_repo`` (clean) + ``docs_user`` +
|
||||
``docs_token`` — DB rows that SHADOW the env (the settings model's
|
||||
standing rule: the database wins over the environment). ``.env`` is
|
||||
never read for writing or rewritten. Idempotent: it runs only while
|
||||
no ``docs_token`` override exists, so owner-set values are never
|
||||
clobbered. One INFO line when it acts (AGENTS rule 10).
|
||||
|
||||
The read side rides the overlay (``load_overrides`` /
|
||||
``resolve_settings`` — the same fail-soft engine); the write side is
|
||||
the PUT route's own write (an ``app_settings`` upsert through the app
|
||||
session — the overlay exposes no write path). A malformed
|
||||
``://user:pass@`` segment (an empty user OR pass) is NOT the legacy
|
||||
shape: the value stays as-is (the pre-phase behavior — nothing breaks,
|
||||
nothing half-migrates), no rows, no log.
|
||||
|
||||
Called ONCE from ``create_app()`` (the composition root, task 04
|
||||
step 2), after the DB is available; a failure there logs a WARNING
|
||||
and never blocks startup — the app boots with the pre-phase behavior
|
||||
(the migration is a convenience, not a dependency).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
|
||||
from sqlalchemy import func
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.settings_overlay import load_overrides, resolve_settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import AppSetting
|
||||
|
||||
logger = logging.getLogger("app.settings")
|
||||
|
||||
#: ``://<user>:<pass>@`` — the legacy credentials-bearing docs-push
|
||||
#: remote (the phase-173 leak class). BOTH groups must be non-empty:
|
||||
#: ``://:@…`` / ``://u:@…`` are malformed, not the legacy shape. The
|
||||
#: same regex the PUT guard uses (``admin_settings._EMBEDDED_CREDS``)
|
||||
#: plus the capture groups; a ``://user:pass@`` subdomain cannot match
|
||||
#: (the character classes exclude ``/`` and the trailing ``@``
|
||||
#: requirement does the job).
|
||||
_EMBEDDED_CREDS = re.compile(r"://([^/@:]+):([^@/]+)@")
|
||||
|
||||
|
||||
def _write_rows(rows: dict[str, str]) -> None:
|
||||
"""Write one row per (key, value) into ``app_settings`` — the PUT
|
||||
route's own write (INSERT … ON CONFLICT (key) DO UPDATE, the row's
|
||||
``updated_at`` moves), one session, one commit. The overlay
|
||||
exposes no write path, so the direct upsert IS the write path.
|
||||
``.env`` is never touched (LOCKED A1 — the DB rows shadow the env;
|
||||
the env file stays the base)."""
|
||||
db: Session = SessionLocal()
|
||||
try:
|
||||
for key, value in rows.items():
|
||||
stmt = pg_insert(AppSetting).values(key=key, value=value)
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=["key"],
|
||||
set_={"value": stmt.excluded.value, "updated_at": func.now()},
|
||||
)
|
||||
db.execute(stmt)
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def migrate_legacy_docs_creds() -> bool:
|
||||
"""The phase-173 C3 migration — split a legacy
|
||||
``://user:pass@`` ``docs_repo`` into three ``app_settings`` rows.
|
||||
|
||||
(1) read the ``app_settings`` overrides (the overlay's state);
|
||||
(2) the target is the EFFECTIVE ``docs_repo`` (an override if
|
||||
present, else the env value — what ``resolve_settings()``
|
||||
resolves to); (3) match ``://<user>:<pass>@`` with BOTH groups
|
||||
non-empty — otherwise no-op (a malformed segment stays as-is, the
|
||||
pre-phase value: nothing breaks, nothing half-migrates); (4) a
|
||||
``docs_token`` override already present → no-op (idempotent —
|
||||
owner-set values are never clobbered, and a previous run of this
|
||||
migration counts as one); (5) otherwise write the three rows
|
||||
(clean repo / user / token — the PUT route's own write) and log
|
||||
ONE INFO line. Returns ``True`` when it acted, ``False`` for
|
||||
every no-op shape. ``.env`` is never read for writing or
|
||||
rewritten: the DB rows shadow the env (the settings model's
|
||||
standing rule).
|
||||
"""
|
||||
effective = resolve_settings().docs_repo
|
||||
match = _EMBEDDED_CREDS.search(effective)
|
||||
if match is None:
|
||||
# (3) clean value OR malformed ``user:pass@`` (empty user or
|
||||
# pass) — the pre-phase value stays as-is; no rows, no log.
|
||||
return False
|
||||
if "docs_token" in load_overrides():
|
||||
# (4) idempotent: a token override (owner-set, or this
|
||||
# migration's own previous run) is never clobbered.
|
||||
return False
|
||||
user, token = match.group(1), match.group(2)
|
||||
# (5) the ``user:pass@`` segment removed, the ``://`` separator
|
||||
# KEPT — ``https://u:pass@host/r.git`` → ``https://host/r.git``.
|
||||
clean = effective.replace(match.group(0), "://", 1)
|
||||
_write_rows(
|
||||
{
|
||||
"docs_repo": clean,
|
||||
"docs_user": user,
|
||||
"docs_token": token,
|
||||
}
|
||||
)
|
||||
# ONE INFO line when it acts (AGENTS rule 10 — the value is never
|
||||
# in the line, A4's log discipline).
|
||||
logger.info(
|
||||
"settings: migrated embedded docs-push credentials out of "
|
||||
"docs_repo into docs_user/docs_token"
|
||||
)
|
||||
return True
|
||||
Reference in New Issue
Block a user