ducoterra
a08a69f143
Two changes, one bug found through the other.
Thinking is policy now: defaultThinkingLevel is xhigh unless you say otherwise
(thinking / --thinking / AISB_THINKING, with pi's levels plus 'host' to decline
and 'off' meaning pi's own no-reasoning level). It is written into the bridged
settings and re-applied by the entrypoint on every run, so a run cannot set its
own reasoning down for the runs after it. Unknown levels are refused with the
list to pick from. Nothing is written for codex, claude or a command after --,
and --no-pi-config declines it along with the bridge, because pi's own defaults
are what that flag asks for.
The reason xhigh had never reached a sandbox is worse than a missing default:
settings.json was managed as one whole file, and pi rewrites its own settings
(a theme change, "seen the changelog"), so the first such write matched nothing
and froze every host preference out permanently -- provider, model, thinking
budgets and level included, a real session left holding just
{"lastChangelogVersion": "1.4.0"}. It is merged per key instead, against a copy
of what we last wrote: a key edited in the sandbox stays edited, a key never
touched follows the host again, an untouched key the host dropped is dropped, a
key pi invented survives, and a file with no base copy takes the host values --
which repairs the volumes already frozen. Unparseable JSON is re-seeded with a
warning since pi cannot use it either; a file the container cannot read is
reported with the fix, because nothing inside the sandbox can repair one.
One trap worth remembering: the seeding script is passed to bash -c in single
quotes, so a single quote inside it closes the argument early and the rest runs
in the outer shell -- valid bash, wrong program, clean bash -n and clean
shellcheck. tests/unit/06_entrypoint.sh extracts the seeding script and asserts
it is quote-free, and extracts the merge program to test its semantics without
a container; tests/e2e/12_sandbox.sh does the same in a real run, including
forcing the level back after the sandbox sets it to off.
LLM-Generated: true
13 KiB
PLAN
Execution plan for ai-sandbox, with the state each phase leaves behind.
Every phase ends with a gate that must pass before the next one starts.
Status as of this writing: phases 1-9 complete, follow-ups listed at the end.
| # | Phase | Gate | Status |
|---|---|---|---|
| 0 | Measure the host before designing | tests/spike_network.sh results recorded in docs/networking.md |
✅ done — killed the sidecar-on-internal-network design, found pasta --map-host-loopback + in-container nftables |
| 1 | Skeleton: CLI, config, preview modes | make test (args, config, precedence) |
✅ done |
| 2 | Image build | build, cache, content-hash tag, contents asserted | ✅ done (tests/e2e/10_image.sh) |
| 3 | Enforcement in the container | generated ruleset shape + real-container capability checks | ✅ done (tests/unit/06_entrypoint.sh, tests/e2e/11_net.sh) |
| 4 | Egress gate | allowlisted host reachable, everything else refused, gate lifecycle | ✅ done (tests/e2e/11_net.sh) |
| 5 | Environment passing | deny-by-default, 0600 delivery, redaction, fail-closed on unset |
✅ done (tests/unit/04_env.sh, tests/e2e/12_sandbox.sh) |
| 6 | Limits, lifecycle, observability | exit codes, cleanup, caps, session log, --keep |
✅ done (tests/e2e/12_sandbox.sh) |
| 7 | Harness defaults, docs, portability sweep | alias UX, README/threat model, shellcheck clean, no personal paths | ✅ done |
| 8 | Supply chain: pinned base, TTL refresh, audit trail | digest in FROM and in the content hash; update/--check; labels + in-image package manifest |
✅ done (tests/unit/07_base.sh, tests/e2e/10_image.sh) |
| 9 | Pi config bridge: same provider inside, credential by env | host models.json/settings.json rewritten not copied; canary secret nowhere in state; seeding, refresh and edit-presence asserted in a real container |
✅ done (tests/unit/08_pi_config.sh, tests/e2e/12_sandbox.sh) |
Phase detail
0 — Measure first (this is why the design looks unusual)
Ran tests/spike_network.sh. Three results decided the architecture:
containers on one user-defined network cannot talk to each other rootless here
(so no sidecar proxy on an internal network); --network=host works rootless and
has internet (so a host-loopback gate is possible); pasta:--map-host-loopback,<addr>
makes a container's connection to <addr> land on the host's loopback; and
--cap-add=NET_ADMIN lets root inside the container install nftables that
genuinely gate the pasta path. Also learned bind mounts need :z under SELinux.
1 — Skeleton
parse_args with the alias-friendly rule (sandbox flags until the first
positional/unknown flag; those become arguments to the harness; -- replaces the
command), config files at two levels, AISB_* overrides, --dry-run /
--print-policy / --print-entrypoint / --print-containerfile.
Precedence is built-in < config file < env < flag, and the built-in defaults live
only in resolve_opts() — materialising them at the top of the script silently
made the environment beat config files, which is a bug worth remembering.
2 — Image
Generated Containerfile, tag = content hash of base image + versions + install
list + entrypoint + Containerfile itself, so a policy change never reuses a stale
image and variants coexist. Node tarballs are checksum-verified against
nodejs.org. pi is baked in so the pi alias works after one build.
3 — Enforcement
entrypoint_script(): read policy → render inet aisb with policy drop →
nft -f → verify the default-drop rule exists → setpriv drop to uid 1000 with
--inh-caps=-all --bounding-set=-all --no-new-privs. Refuses to start (77/78)
rather than run unenforced.
4 — Gate
Per-session squid container on --network=host, bound to 127.0.0.1:<free port>,
config generated and bind-mounted read-only, http_access deny all last.
Sandbox gets proxy env + a ruleset that allows nothing else out. Torn down with
the session unless gate up created a named one.
5-7 — Environment, lifecycle, polish
0600 env file delivery and redaction everywhere; limits and caps asserted from
inside; --keep leaves the container and the session dir for forensics;
clean reaps (--all for running ones, --volumes[=GLOB] for unused home caches,
never a volume a container still references); doctor verifies the host including a
real bind/uid round-trip; docs; shellcheck-clean in a container.
/home/sandbox is a named volume so npm and tool caches survive between runs. It
outlives the image, which is the one kind of staleness this design cannot hash away:
change the image's home layout and existing volumes keep the old files, so
ensure_home_volume() creates it labelled and clean --volumes is the documented way
to drop it.
8 — Supply chain
The hash made our changes traceable but said nothing about the base: FROM ubuntu:24.04 is a moving pointer, podman build's pull policy is missing, and
ensure_image() returns early when the tag exists — so even build --rebuild reused
the cached base. Now the tag is resolved to a digest (skopeo, falling back to
podman pull unless AISB_BASE_PULL_FALLBACK=0), recorded per-ref under
$AISB_CACHE_DIR/base/, and used in FROM and in image_hash(): a refreshed base
yields a new image rather than mutating one that may be running.
Two rules make it survivable rather than merely strict. A stale pin beats a mutable
tag — offline, or with AISB_BASE_AUTO_UPDATE=0, the recorded digest is used, and
only a build with no digest at all is allowed, loudly marked UNPINNED in the output,
in the image labels and in doctor. Freshness is a TTL (AISB_BASE_TTL_DAYS=7,
0 every build, -1 never) rather than a background job, so nothing surprises you
mid-session and update stays an explicit, dated event. Both images carry
aisb.base-digest/aisb.node/aisb.pi-version labels and
/etc/aisb/manifest.txt with resolved apt versions, which is how you answer "which
squid was enforcing this?" a month later. --print-* stays cache-only and
deterministic; inspect is the surface that asks the machine now.
One bug worth remembering: POL[base_digest]="$(base_pin …)" looked right and
silently produced unpinned labels, because the subshell's assignment was discarded.
Helpers return text; callers assign.
9 — Pi configuration bridge
The image ships pi with no provider and no preferences, so the first run asked you to
configure the model twice. It now reads ~/.pi/agent/{models,settings}.json and
rewrites them: every provider apiKey becomes "$AI_API_TOKEN" (pi resolves an env
reference itself, so no value is ever written to disk), !command header values are
dropped rather than executed, settings pointing into $HOME are dropped because that
path is not mounted, and auth.json/models-store.json/sessions/extensions are never
read. The pair lands in /home/sandbox/.pi/agent at container start, owned by uid 1000,
mode 600, and is a managed file: a host change is re-seeded, a sandbox edit is kept and
reported, a deletion takes the host version again.
Two decisions worth keeping. The token follows the existing env rules — one variable,
passed by name through the 0600 file, announced when auto-passed — and a bridged run
where that variable is unset or empty is refused instead of starting a pi that cannot
reach its model, because "sandbox is broken" is a worse bug report than an explicit
refusal. And the bridge never widens the allowlist; it warns with the exact --allow
to add, since an agent silently unable to answer looks like a network failure.
10 — The .agents share
A sandboxed harness found none of your skills, prompt templates or standing AGENTS.md,
because those live in ~/.agents and only the project is mounted. It is now bind-mounted
read-only at /home/sandbox/.agents by default (agents, agents_dir, agents_dest;
--agents, --no-agents, --agents-dir), which is where pi looks for user-level skills.
Bound, never copied, so the sandbox always sees the live tree and nothing of it lands in
ai-sandbox state.
Read-only is the load-bearing part and there is deliberately no way to lift it: a skill is
an instruction the model follows later, on the host, so a sandbox that could write
~/.agents had a persistence path out of itself, planted as a helpful-looking
SKILL.md. It holds even under --insecure-keep-root (e2e-checked), the mount is nested
inside the home volume after it so it cannot be shadowed, and an explicit --mount on
the same destination wins because podman refuses duplicates. A missing or unreadable
directory is a silent skip like the pi bridge's; an explicit --agents-dir that resolves
to nothing is a hard error.
11 — Thinking level as policy, and a settings merge that does not freeze
defaultThinkingLevel is now xhigh by default (thinking, --thinking,
AISB_THINKING; pi's own default is medium), written into the bridged settings and
re-applied by the entrypoint on every run, so a run cannot lower the reasoning of the runs
after it. --thinking host hands the key back to settings.json, off is pi's own
"no reasoning" level, and an unknown value is refused with the list to pick from. Nothing
is written for codex, claude or a command after --, and --no-pi-config declines the
policy along with the bridge.
Getting there found a worse bug. settings.json was managed as one managed file, and pi
rewrites its own settings (a theme change, "seen the changelog"), so the first such write
matched nothing and froze every host preference out of the sandbox for good -- provider, model,
thinking budgets and the level included, with a real session left holding
{"lastChangelogVersion": "1.4.0"}. It is merged per key instead, against a copy of what
we last wrote (.aisb-settings.json.wrote): sandbox edits win per key, untouched keys
follow the host again, untouched keys the host dropped are dropped, keys pi invented
survive, and a file that is missing a base copy takes the host values -- which is what
repairs the volumes already frozen. Unparseable JSON is re-seeded with a warning (pi
cannot use it either); a file the container cannot read is reported with the fix, because
nothing inside the sandbox can repair it.
One trap worth remembering: the seeding script is passed to bash -c in single quotes, so
a single quote anywhere inside it closes the argument early and the remainder becomes code
in the outer shell -- valid bash, wrong program, invisible to bash -n and shellcheck.
tests/unit/06_entrypoint.sh now extracts the seeding script and asserts it is quote-free,
and extracts the merge program to test its semantics without a container.
Follow-ups (not yet done)
--net unixmode:--network=noneplus a unix-socket proxy and an in-container TCP→socket forwarder. Structurally stronger (no IP stack at all); needs a story for tools that resolve names themselves.- Re-resolve
--no-gatedomains on a timer, or resolve through the gate and pin the returned answers, so long-running sessions do not go stale against CDNs. - Idle gate reaping beyond
clean: a--gate-ttllabel check on start. --allow-from-lockfile: derive a first-cut allowlist frompackage.json/requirements.txtregistries so the first run is not a blank page.- Quarantine mode for untrusted repos:
--net none+ read-only mount +--ephemeral-homeas a single flag, plus a warning if.githooks exist. --record: optional pcap/CONNECT-log capture of a session for review.- Packaging: a
quadlet/systemd unit for a long-lived named gate. - Signature verification for the base where the registry signs it (cosign/sigstore),
so a pin is authenticated rather than merely immutable; and an optional local
mirror/
AISB_SKOPEO-compatible offline source for air-gapped hosts. update --autoas a timer unit, if TTL-on-next-run turns out to be too lazy for security response — deliberately not done yet, since a background rebuild that surprises a running session is its own hazard.- Scoped sandbox tokens: if the gateway can mint a short-lived, per-sandbox key,
--pi-token-envshould ask for one rather than forwarding the long-lived host credential. The bridge deliberately does not guess a minting endpoint. --thinkingfor the other harnesses: codex and claude have their own reasoning knobs; a policy that spans harnesses needs to know each one's config shape rather than assume pi's.- Bridging extensions/skills on request:
--pi-mount-extor similar for the extension paths named inenabledExtensionPaths, which are host paths and therefore currently dropped; needs the same explicit-grant shape as--mount. models.jsonreview surface:inspectcould print the exact diff between host config and bridged config, so a change is reviewable before a run rather than after.