ducoterra
a08a69f143
Two changes, one bug found through the other.
Thinking is policy now: defaultThinkingLevel is xhigh unless you say otherwise
(thinking / --thinking / AISB_THINKING, with pi's levels plus 'host' to decline
and 'off' meaning pi's own no-reasoning level). It is written into the bridged
settings and re-applied by the entrypoint on every run, so a run cannot set its
own reasoning down for the runs after it. Unknown levels are refused with the
list to pick from. Nothing is written for codex, claude or a command after --,
and --no-pi-config declines it along with the bridge, because pi's own defaults
are what that flag asks for.
The reason xhigh had never reached a sandbox is worse than a missing default:
settings.json was managed as one whole file, and pi rewrites its own settings
(a theme change, "seen the changelog"), so the first such write matched nothing
and froze every host preference out permanently -- provider, model, thinking
budgets and level included, a real session left holding just
{"lastChangelogVersion": "1.4.0"}. It is merged per key instead, against a copy
of what we last wrote: a key edited in the sandbox stays edited, a key never
touched follows the host again, an untouched key the host dropped is dropped, a
key pi invented survives, and a file with no base copy takes the host values --
which repairs the volumes already frozen. Unparseable JSON is re-seeded with a
warning since pi cannot use it either; a file the container cannot read is
reported with the fix, because nothing inside the sandbox can repair one.
One trap worth remembering: the seeding script is passed to bash -c in single
quotes, so a single quote inside it closes the argument early and the rest runs
in the outer shell -- valid bash, wrong program, clean bash -n and clean
shellcheck. tests/unit/06_entrypoint.sh extracts the seeding script and asserts
it is quote-free, and extracts the merge program to test its semantics without
a container; tests/e2e/12_sandbox.sh does the same in a real run, including
forcing the level back after the sandbox sets it to off.
LLM-Generated: true
188 lines
13 KiB
Markdown
188 lines
13 KiB
Markdown
# PLAN
|
|
|
|
Execution plan for `ai-sandbox`, with the state each phase leaves behind.
|
|
Every phase ends with a gate that must pass before the next one starts.
|
|
Status as of this writing: **phases 1-9 complete**, follow-ups listed at the end.
|
|
|
|
| # | Phase | Gate | Status |
|
|
|---|-------|------|--------|
|
|
| 0 | Measure the host before designing | `tests/spike_network.sh` results recorded in `docs/networking.md` | ✅ done — killed the sidecar-on-internal-network design, found `pasta --map-host-loopback` + in-container nftables |
|
|
| 1 | Skeleton: CLI, config, preview modes | `make test` (args, config, precedence) | ✅ done |
|
|
| 2 | Image build | build, cache, content-hash tag, contents asserted | ✅ done (`tests/e2e/10_image.sh`) |
|
|
| 3 | Enforcement in the container | generated ruleset shape + real-container capability checks | ✅ done (`tests/unit/06_entrypoint.sh`, `tests/e2e/11_net.sh`) |
|
|
| 4 | Egress gate | allowlisted host reachable, everything else refused, gate lifecycle | ✅ done (`tests/e2e/11_net.sh`) |
|
|
| 5 | Environment passing | deny-by-default, `0600` delivery, redaction, fail-closed on unset | ✅ done (`tests/unit/04_env.sh`, `tests/e2e/12_sandbox.sh`) |
|
|
| 6 | Limits, lifecycle, observability | exit codes, cleanup, caps, session log, `--keep` | ✅ done (`tests/e2e/12_sandbox.sh`) |
|
|
| 7 | Harness defaults, docs, portability sweep | alias UX, README/threat model, shellcheck clean, no personal paths | ✅ done |
|
|
| 8 | Supply chain: pinned base, TTL refresh, audit trail | digest in `FROM` and in the content hash; `update`/`--check`; labels + in-image package manifest | ✅ done (`tests/unit/07_base.sh`, `tests/e2e/10_image.sh`) |
|
|
| 9 | Pi config bridge: same provider inside, credential by env | host `models.json`/`settings.json` rewritten not copied; canary secret nowhere in state; seeding, refresh and edit-presence asserted in a real container | ✅ done (`tests/unit/08_pi_config.sh`, `tests/e2e/12_sandbox.sh`) |
|
|
|
|
## Phase detail
|
|
|
|
### 0 — Measure first (this is why the design looks unusual)
|
|
Ran `tests/spike_network.sh`. Three results decided the architecture:
|
|
containers on one user-defined network cannot talk to each other rootless here
|
|
(so no sidecar proxy on an internal network); `--network=host` works rootless and
|
|
has internet (so a host-loopback gate is possible); `pasta:--map-host-loopback,<addr>`
|
|
makes a container's connection to `<addr>` land on the host's loopback; and
|
|
`--cap-add=NET_ADMIN` lets root *inside* the container install nftables that
|
|
genuinely gate the pasta path. Also learned bind mounts need `:z` under SELinux.
|
|
|
|
### 1 — Skeleton
|
|
`parse_args` with the alias-friendly rule (sandbox flags until the first
|
|
positional/unknown flag; those become arguments to the harness; `--` replaces the
|
|
command), config files at two levels, `AISB_*` overrides, `--dry-run` /
|
|
`--print-policy` / `--print-entrypoint` / `--print-containerfile`.
|
|
Precedence is built-in < config file < env < flag, and the built-in defaults live
|
|
only in `resolve_opts()` — materialising them at the top of the script silently
|
|
made the environment beat config files, which is a bug worth remembering.
|
|
|
|
### 2 — Image
|
|
Generated Containerfile, tag = content hash of base image + versions + install
|
|
list + entrypoint + Containerfile itself, so a policy change never reuses a stale
|
|
image and variants coexist. Node tarballs are checksum-verified against
|
|
nodejs.org. `pi` is baked in so the `pi` alias works after one build.
|
|
|
|
### 3 — Enforcement
|
|
`entrypoint_script()`: read policy → render `inet aisb` with `policy drop` →
|
|
`nft -f` → verify the default-drop rule exists → `setpriv` drop to uid 1000 with
|
|
`--inh-caps=-all --bounding-set=-all --no-new-privs`. Refuses to start (77/78)
|
|
rather than run unenforced.
|
|
|
|
### 4 — Gate
|
|
Per-session squid container on `--network=host`, bound to `127.0.0.1:<free port>`,
|
|
config generated and bind-mounted read-only, `http_access deny all` last.
|
|
Sandbox gets proxy env + a ruleset that allows nothing else out. Torn down with
|
|
the session unless `gate up` created a named one.
|
|
|
|
### 5-7 — Environment, lifecycle, polish
|
|
`0600` env file delivery and redaction everywhere; limits and caps asserted from
|
|
inside; `--keep` leaves the container *and* the session dir for forensics;
|
|
`clean` reaps (`--all` for running ones, `--volumes[=GLOB]` for unused home caches,
|
|
never a volume a container still references); `doctor` verifies the host including a
|
|
real bind/uid round-trip; docs; shellcheck-clean in a container.
|
|
|
|
`/home/sandbox` is a named volume so npm and tool caches survive between runs. It
|
|
outlives the image, which is the one kind of staleness this design cannot hash away:
|
|
change the image's home layout and existing volumes keep the old files, so
|
|
`ensure_home_volume()` creates it labelled and `clean --volumes` is the documented way
|
|
to drop it.
|
|
|
|
### 8 — Supply chain
|
|
The hash made *our* changes traceable but said nothing about the base: `FROM
|
|
ubuntu:24.04` is a moving pointer, `podman build`'s pull policy is *missing*, and
|
|
`ensure_image()` returns early when the tag exists — so even `build --rebuild` reused
|
|
the cached base. Now the tag is resolved to a digest (`skopeo`, falling back to
|
|
`podman pull` unless `AISB_BASE_PULL_FALLBACK=0`), recorded per-ref under
|
|
`$AISB_CACHE_DIR/base/`, and used in `FROM` *and* in `image_hash()`: a refreshed base
|
|
yields a new image rather than mutating one that may be running.
|
|
|
|
Two rules make it survivable rather than merely strict. **A stale pin beats a mutable
|
|
tag** — offline, or with `AISB_BASE_AUTO_UPDATE=0`, the recorded digest is used, and
|
|
only a build with no digest at all is allowed, loudly marked `UNPINNED` in the output,
|
|
in the image labels and in `doctor`. **Freshness is a TTL** (`AISB_BASE_TTL_DAYS=7`,
|
|
`0` every build, `-1` never) rather than a background job, so nothing surprises you
|
|
mid-session and `update` stays an explicit, dated event. Both images carry
|
|
`aisb.base-digest`/`aisb.node`/`aisb.pi-version` labels and
|
|
`/etc/aisb/manifest.txt` with resolved apt versions, which is how you answer "which
|
|
squid was enforcing this?" a month later. `--print-*` stays cache-only and
|
|
deterministic; `inspect` is the surface that asks the machine now.
|
|
|
|
One bug worth remembering: `POL[base_digest]="$(base_pin …)"` looked right and
|
|
silently produced `unpinned` labels, because the subshell's assignment was discarded.
|
|
Helpers return text; callers assign.
|
|
|
|
### 9 — Pi configuration bridge
|
|
The image ships `pi` with no provider and no preferences, so the first run asked you to
|
|
configure the model twice. It now reads `~/.pi/agent/{models,settings}.json` and
|
|
rewrites them: every provider `apiKey` becomes `"$AI_API_TOKEN"` (pi resolves an env
|
|
reference itself, so no value is ever written to disk), `!command` header values are
|
|
dropped rather than executed, settings pointing into `$HOME` are dropped because that
|
|
path is not mounted, and `auth.json`/`models-store.json`/sessions/extensions are never
|
|
read. The pair lands in `/home/sandbox/.pi/agent` at container start, owned by uid 1000,
|
|
mode 600, and is a managed file: a host change is re-seeded, a sandbox edit is kept and
|
|
reported, a deletion takes the host version again.
|
|
|
|
Two decisions worth keeping. The token follows the existing env rules — one variable,
|
|
passed by name through the `0600` file, announced when auto-passed — and a bridged run
|
|
where that variable is unset *or empty* is refused instead of starting a pi that cannot
|
|
reach its model, because "sandbox is broken" is a worse bug report than an explicit
|
|
refusal. And the bridge never widens the allowlist; it warns with the exact `--allow`
|
|
to add, since an agent silently unable to answer looks like a network failure.
|
|
|
|
### 10 — The `.agents` share
|
|
A sandboxed harness found none of your skills, prompt templates or standing `AGENTS.md`,
|
|
because those live in `~/.agents` and only the project is mounted. It is now bind-mounted
|
|
read-only at `/home/sandbox/.agents` by default (`agents`, `agents_dir`, `agents_dest`;
|
|
`--agents`, `--no-agents`, `--agents-dir`), which is where pi looks for user-level skills.
|
|
Bound, never copied, so the sandbox always sees the live tree and nothing of it lands in
|
|
ai-sandbox state.
|
|
|
|
Read-only is the load-bearing part and there is deliberately no way to lift it: a skill is
|
|
an instruction the model follows *later*, on the host, so a sandbox that could write
|
|
`~/.agents` had a persistence path out of itself, planted as a helpful-looking
|
|
`SKILL.md`. It holds even under `--insecure-keep-root` (e2e-checked), the mount is nested
|
|
inside the home volume *after* it so it cannot be shadowed, and an explicit `--mount` on
|
|
the same destination wins because podman refuses duplicates. A missing or unreadable
|
|
directory is a silent skip like the pi bridge's; an explicit `--agents-dir` that resolves
|
|
to nothing is a hard error.
|
|
|
|
### 11 — Thinking level as policy, and a settings merge that does not freeze
|
|
`defaultThinkingLevel` is now `xhigh` by default (`thinking`, `--thinking`,
|
|
`AISB_THINKING`; pi's own default is `medium`), written into the bridged settings and
|
|
re-applied by the entrypoint on every run, so a run cannot lower the reasoning of the runs
|
|
after it. `--thinking host` hands the key back to `settings.json`, `off` is pi's own
|
|
"no reasoning" level, and an unknown value is refused with the list to pick from. Nothing
|
|
is written for codex, claude or a command after `--`, and `--no-pi-config` declines the
|
|
policy along with the bridge.
|
|
|
|
Getting there found a worse bug. `settings.json` was managed as one managed file, and pi
|
|
rewrites its own settings (a theme change, "seen the changelog"), so the first such write
|
|
matched nothing and froze *every* host preference out of the sandbox for good -- provider, model,
|
|
thinking budgets and the level included, with a real session left holding
|
|
`{"lastChangelogVersion": "1.4.0"}`. It is merged per key instead, against a copy of what
|
|
we last wrote (`.aisb-settings.json.wrote`): sandbox edits win per key, untouched keys
|
|
follow the host again, untouched keys the host dropped are dropped, keys pi invented
|
|
survive, and a file that is missing a base copy takes the host values -- which is what
|
|
repairs the volumes already frozen. Unparseable JSON is re-seeded with a warning (pi
|
|
cannot use it either); a file the container cannot *read* is reported with the fix, because
|
|
nothing inside the sandbox can repair it.
|
|
|
|
One trap worth remembering: the seeding script is passed to `bash -c` in single quotes, so
|
|
a single quote anywhere inside it closes the argument early and the remainder becomes code
|
|
in the outer shell -- valid bash, wrong program, invisible to `bash -n` and shellcheck.
|
|
`tests/unit/06_entrypoint.sh` now extracts the seeding script and asserts it is quote-free,
|
|
and extracts the merge program to test its semantics without a container.
|
|
|
|
## Follow-ups (not yet done)
|
|
|
|
* **`--net unix` mode**: `--network=none` plus a unix-socket proxy and an in-container
|
|
TCP→socket forwarder. Structurally stronger (no IP stack at all); needs a story
|
|
for tools that resolve names themselves.
|
|
* **Re-resolve `--no-gate` domains** on a timer, or resolve through the gate and pin
|
|
the returned answers, so long-running sessions do not go stale against CDNs.
|
|
* **Idle gate reaping** beyond `clean`: a `--gate-ttl` label check on start.
|
|
* **`--allow-from-lockfile`**: derive a first-cut allowlist from
|
|
`package.json`/`requirements.txt` registries so the first run is not a blank page.
|
|
* **Quarantine mode for untrusted repos**: `--net none` + read-only mount +
|
|
`--ephemeral-home` as a single flag, plus a warning if `.git` hooks exist.
|
|
* **`--record`**: optional pcap/CONNECT-log capture of a session for review.
|
|
* Packaging: a `quadlet`/systemd unit for a long-lived named gate.
|
|
* Signature verification for the base where the registry signs it (cosign/sigstore),
|
|
so a pin is authenticated rather than merely immutable; and an optional local
|
|
mirror/`AISB_SKOPEO`-compatible offline source for air-gapped hosts.
|
|
* `update --auto` as a timer unit, if TTL-on-next-run turns out to be too lazy for
|
|
security response — deliberately not done yet, since a background rebuild that
|
|
surprises a running session is its own hazard.
|
|
* **Scoped sandbox tokens**: if the gateway can mint a short-lived, per-sandbox key,
|
|
`--pi-token-env` should ask for one rather than forwarding the long-lived host
|
|
credential. The bridge deliberately does not guess a minting endpoint.
|
|
* **`--thinking` for the other harnesses**: codex and claude have their own reasoning
|
|
knobs; a policy that spans harnesses needs to know each one's config shape rather than
|
|
assume pi's.
|
|
* **Bridging extensions/skills on request**: `--pi-mount-ext` or similar for the
|
|
extension paths named in `enabledExtensionPaths`, which are host paths and therefore
|
|
currently dropped; needs the same explicit-grant shape as `--mount`.
|
|
* **`models.json` review surface**: `inspect` could print the exact diff between host
|
|
config and bridged config, so a change is reviewable before a run rather than after.
|