make xhigh the sandbox's thinking, and stop settings.json freezing
Two changes, one bug found through the other.
Thinking is policy now: defaultThinkingLevel is xhigh unless you say otherwise
(thinking / --thinking / AISB_THINKING, with pi's levels plus 'host' to decline
and 'off' meaning pi's own no-reasoning level). It is written into the bridged
settings and re-applied by the entrypoint on every run, so a run cannot set its
own reasoning down for the runs after it. Unknown levels are refused with the
list to pick from. Nothing is written for codex, claude or a command after --,
and --no-pi-config declines it along with the bridge, because pi's own defaults
are what that flag asks for.
The reason xhigh had never reached a sandbox is worse than a missing default:
settings.json was managed as one whole file, and pi rewrites its own settings
(a theme change, "seen the changelog"), so the first such write matched nothing
and froze every host preference out permanently -- provider, model, thinking
budgets and level included, a real session left holding just
{"lastChangelogVersion": "1.4.0"}. It is merged per key instead, against a copy
of what we last wrote: a key edited in the sandbox stays edited, a key never
touched follows the host again, an untouched key the host dropped is dropped, a
key pi invented survives, and a file with no base copy takes the host values --
which repairs the volumes already frozen. Unparseable JSON is re-seeded with a
warning since pi cannot use it either; a file the container cannot read is
reported with the fix, because nothing inside the sandbox can repair one.
One trap worth remembering: the seeding script is passed to bash -c in single
quotes, so a single quote inside it closes the argument early and the rest runs
in the outer shell -- valid bash, wrong program, clean bash -n and clean
shellcheck. tests/unit/06_entrypoint.sh extracts the seeding script and asserts
it is quote-free, and extracts the merge program to test its semantics without
a container; tests/e2e/12_sandbox.sh does the same in a real run, including
forcing the level back after the sandbox sets it to off.
LLM-Generated: true
This commit is contained in:
@@ -245,6 +245,55 @@ sb $DARK --harness pi -- bash -c 'rm -f "$HOME/.pi/agent/models.json"; echo rese
|
||||
sb $DARK --harness pi -- bash -lc 'grep -q 524288 "$HOME/.pi/agent/models.json" && echo reseeded || echo not_reseeded'
|
||||
assert_contains "reseeded" "deleting it inside the sandbox takes the host version again"
|
||||
|
||||
t_name "settings.json follows the host per key, and the thinking level is policy"
|
||||
# The host file says medium and no ai-sandbox policy is in it: anything that arrives as
|
||||
# xhigh can only have been written by the policy.
|
||||
printf '{ "defaultProvider": "gateway", "defaultModel": "turbo", "theme": "dark/dark",\n "defaultThinkingLevel": "medium", "thinkingBudgets": { "xhigh": 65536 } }\n' > "$AGENT/settings.json"
|
||||
sb $DARK --harness pi -- bash -lc 'jq -c . "$HOME/.pi/agent/settings.json"'
|
||||
assert_contains '"defaultThinkingLevel":"xhigh"' "xhigh is enforced over the host file's medium"
|
||||
assert_contains '"thinkingBudgets":{"xhigh":65536}' "and the rest of the preferences come along"
|
||||
|
||||
# What pi does to its own settings: rewrites the file, adding a key of its own. That used
|
||||
# to freeze every host preference out of the sandbox for good.
|
||||
sb $DARK --harness pi -- bash -c '
|
||||
f="$HOME/.pi/agent/settings.json"
|
||||
jq ".theme=\"light/light\" | .defaultThinkingLevel=\"off\" | .lastChangelogVersion=\"9.9.9\"" "$f" > "$f.n" && mv "$f.n" "$f"
|
||||
echo edited' >/dev/null
|
||||
# the host then moves the model and the budgets, drops the provider entirely, and says
|
||||
# medium itself -- so a later --thinking host run has something to hand over
|
||||
printf '{ "defaultModel": "deepthink", "theme": "sepia/sepia",\n "defaultThinkingLevel": "medium", "thinkingBudgets": { "xhigh": 131072 } }\n' > "$AGENT/settings.json"
|
||||
sb $DARK --harness pi -- bash -lc 'jq -c . "$HOME/.pi/agent/settings.json"'
|
||||
MERGED="$OUT"
|
||||
printf '%s' "$MERGED" | grep -qF '"theme":"light/light"' && _ok "the theme edited in the sandbox survives" \
|
||||
|| _bad "the theme edited in the sandbox survives" "$MERGED"
|
||||
printf '%s' "$MERGED" | grep -qF '"lastChangelogVersion":"9.9.9"' && _ok "a key only pi wrote survives" \
|
||||
|| _bad "a key only pi wrote survives" "$MERGED"
|
||||
printf '%s' "$MERGED" | grep -qF '"defaultModel":"deepthink"' && _ok "an untouched model follows the host again" \
|
||||
|| _bad "an untouched model follows the host again" "$MERGED"
|
||||
printf '%s' "$MERGED" | grep -qF '"xhigh":131072' && _ok "and so do the thinking budgets" \
|
||||
|| _bad "and so do the thinking budgets" "$MERGED"
|
||||
printf '%s' "$MERGED" | grep -qF 'defaultProvider' && _bad "an untouched key the host dropped is dropped" \
|
||||
|| _ok "an untouched key the host dropped is dropped"
|
||||
printf '%s' "$MERGED" | grep -qF '"defaultThinkingLevel":"xhigh"' && _ok "thinking is forced back to xhigh after the sandbox set off" \
|
||||
|| _bad "thinking is forced back to xhigh after the sandbox set off" "$MERGED"
|
||||
|
||||
# (the jq programs below avoid `{a, b}` object construction: bash brace-expands an
|
||||
# unquoted word containing a comma before jq ever sees it)
|
||||
sb $DARK --harness pi --thinking host -- bash -lc 'jq -r .defaultThinkingLevel "$HOME/.pi/agent/settings.json"'
|
||||
assert_eq "medium" "$(printf '%s' "$OUT" | tail -1)" "with --thinking host the host's own medium is handed over"
|
||||
|
||||
sb $DARK --harness pi -- bash -c 'printf "{oops" > "$HOME/.pi/agent/settings.json"' >/dev/null
|
||||
sb $DARK --harness pi -- bash -lc 'jq -c . "$HOME/.pi/agent/settings.json"'
|
||||
assert_contains "deepthink" "a settings.json that is not JSON is re-seeded rather than left broken"
|
||||
assert_contains '"defaultThinkingLevel":"xhigh"' "with the policy still applied"
|
||||
sb $DARK --harness pi -- bash -lc 'jq -r .defaultThinkingLevel "$HOME/.pi/agent/settings.json"'
|
||||
assert_eq "xhigh" "$(printf '%s' "$OUT" | tail -1)" "and it stays xhigh, not the medium that came back"
|
||||
|
||||
sb $DARK --harness pi -- bash -c 'chmod 000 "$HOME/.pi/agent/settings.json"' >/dev/null
|
||||
sb $DARK --harness pi -- bash -lc 'echo ran'
|
||||
assert_contains "cannot read settings.json" "a file this container cannot read is reported, not blamed on JSON"
|
||||
assert_contains "--ephemeral-home" "and the report names a way out"
|
||||
|
||||
unset AI_API_TOKEN
|
||||
sb_timeout 90 $DARK --harness pi
|
||||
assert_rc 1 "a run without the credential is refused before anything starts"
|
||||
|
||||
+1
-1
@@ -79,7 +79,7 @@ for f in "${FILES[@]}"; do
|
||||
# test should depend on or accidentally share. Files that exercise the .agents
|
||||
# bridge point AISB_AGENTS_DIR at a fixture of their own.
|
||||
export AISB_AGENTS_DIR="${AISB_AGENTS_DIR:-$W/absent-agents-dir}"
|
||||
unset AI_API_TOKEN AISB_PI_TOKEN_ENV AISB_PI_CONFIG AISB_PI_TOKEN_AUTO
|
||||
unset AI_API_TOKEN AISB_PI_TOKEN_ENV AISB_PI_CONFIG AISB_PI_TOKEN_AUTO AISB_THINKING
|
||||
# NOTE: HOME is deliberately not overridden -- podman storage lives there.
|
||||
unset AISB_ALLOW AISB_ENV AISB_MOUNT AISB_INSTALL AISB_NET AISB_EXTRA_ARGS AISB_SESSION_OVERRIDE
|
||||
unset AISB_AGENTS AISB_AGENTS_DEST
|
||||
|
||||
@@ -61,6 +61,83 @@ grep -c 'accept comment "aisb-direct"' "$RULES" | grep -qx 3 && _ok "exactly 3 d
|
||||
t_name "no DNS hole when the gate is in front"
|
||||
grep -E 'th dport 53' "$RULES" >/dev/null && _bad "DNS is closed in gate mode" || _ok "DNS is closed in gate mode"
|
||||
|
||||
t_name "the seeding script survives being passed in single quotes"
|
||||
# Everything between `bash -c '` and `' aisb-seed` is one single-quoted argument of the
|
||||
# outer script. One quote inside it closes that argument early and the rest of the file
|
||||
# silently becomes code in the *outer* shell -- valid bash, wrong program, so neither
|
||||
# `bash -n` nor shellcheck notices. Hence this test.
|
||||
INNER="$WORK/seeding-inner.sh"
|
||||
awk "/setpriv .*bash -c '/{f=1;next} /^ *' aisb-seed/{f=0} f" "$EP" > "$INNER"
|
||||
[[ -s "$INNER" ]] && _ok "the seeding script was extracted" || _bad "the seeding script was extracted"
|
||||
if grep -q "'" "$INNER"; then
|
||||
_bad "it contains no single quote" "found: $(grep -n "'" "$INNER" | head -2 | tr '\n' ' ')"
|
||||
else
|
||||
_ok "it contains no single quote"
|
||||
fi
|
||||
bash -n "$INNER" && _ok "and it is valid bash on its own" || _bad "and it is valid bash on its own"
|
||||
grep -q '\$lvl' "$INNER" && _ok "the thinking level is enforced inside it" || _bad "thinking level enforced"
|
||||
|
||||
t_name "settings.json is merged per key, not managed whole"
|
||||
# Run the merge exactly as the entrypoint does, with the program extracted from the image
|
||||
# script: T = what the sandbox has, B = what we last wrote, N = what the host says now.
|
||||
MJQ="$WORK/merge.jq"
|
||||
awk '/^SETTINGS_MERGE_JQ=/{f=1;next} f && /^.$/{exit} f' "$EP" > "$MJQ"
|
||||
[[ "$(wc -l < "$MJQ")" -gt 3 ]] && _ok "the merge program was extracted" || _bad "the merge program was extracted"
|
||||
M_IN="$WORK/m"; mkdir -p "$M_IN"
|
||||
cat > "$M_IN/new.json" <<'EOF'
|
||||
{ "defaultProvider": "gateway", "defaultModel": "turbo", "theme": "dark/dark",
|
||||
"thinkingBudgets": { "xhigh": 65536 } }
|
||||
EOF
|
||||
cat > "$M_IN/wrote.json" <<'EOF'
|
||||
{ "defaultProvider": "gateway", "defaultModel": "turbo", "theme": "dark/dark",
|
||||
"thinkingBudgets": { "xhigh": 65536 } }
|
||||
EOF
|
||||
cat > "$M_IN/sandbox.json" <<'EOF'
|
||||
{ "defaultProvider": "gateway", "defaultModel": "turbo", "theme": "light/light",
|
||||
"thinkingBudgets": { "xhigh": 65536 }, "lastChangelogVersion": "1.4.0", "mine": 7 }
|
||||
EOF
|
||||
merge_out() { jq -c -n --slurpfile T "$1" --slurpfile B "$2" --slurpfile N "$3" -f "$MJQ"; }
|
||||
M="$(merge_out "$M_IN/sandbox.json" "$M_IN/wrote.json" "$M_IN/new.json")"
|
||||
[[ "$(printf '%s' "$M" | jq -r .theme)" == "light/light" ]] \
|
||||
&& _ok "a key edited in the sandbox stays edited" || _bad "sandbox edit kept: $M"
|
||||
[[ "$(printf '%s' "$M" | jq -r .lastChangelogVersion)" == "1.4.0" ]] \
|
||||
&& _ok "a key only pi wrote survives" || _bad "sandbox-only key: $M"
|
||||
[[ "$(printf '%s' "$M" | jq -r .mine)" == "7" ]] \
|
||||
&& _ok "and so does one the sandbox invented" || _bad "invented key: $M"
|
||||
|
||||
# host moves two keys, drops a third, keeps one the sandbox changed
|
||||
cat > "$M_IN/new2.json" <<'EOF'
|
||||
{ "defaultProvider": "gateway", "defaultModel": "deepthink",
|
||||
"thinkingBudgets": { "xhigh": 131072 } }
|
||||
EOF
|
||||
M2="$(merge_out "$M_IN/sandbox.json" "$M_IN/wrote.json" "$M_IN/new2.json")"
|
||||
[[ "$(printf '%s' "$M2" | jq -r .defaultModel)" == "deepthink" ]] \
|
||||
&& _ok "an untouched key follows the host again" || _bad "host key arrives: $M2"
|
||||
[[ "$(printf '%s' "$M2" | jq -r .thinkingBudgets.xhigh)" == "131072" ]] \
|
||||
&& _ok "including a nested one (whole key, whole value)" || _bad "nested key: $M2"
|
||||
[[ "$(printf '%s' "$M2" | jq -r 'has("theme")')" == "true" ]] \
|
||||
&& _ok "a key the host dropped stays if the sandbox had changed it" || _bad "dropped-but-edited key: $M2"
|
||||
# the same drop against a file the sandbox never touched: it goes away, as it did when
|
||||
# the whole file was re-seeded -- otherwise settings only ever accumulate
|
||||
cat > "$M_IN/clean.json" <<'EOF'
|
||||
{ "defaultProvider": "gateway", "defaultModel": "turbo", "theme": "dark/dark",
|
||||
"thinkingBudgets": { "xhigh": 65536 } }
|
||||
EOF
|
||||
M3="$(merge_out "$M_IN/clean.json" "$M_IN/wrote.json" "$M_IN/new2.json")"
|
||||
[[ "$(printf '%s' "$M3" | jq -r 'has("theme")')" == "false" ]] \
|
||||
&& _ok "an untouched key the host dropped is dropped" || _bad "untouched dropped key: $M3"
|
||||
# no base copy (the first run after this shipped, or .wrote deleted): the host wins on
|
||||
# every key it has, which is what repairs a settings file pi had frozen.
|
||||
M4="$(merge_out "$M_IN/sandbox.json" /dev/null "$M_IN/new.json")"
|
||||
[[ "$(printf '%s' "$M4" | jq -r '.thinkingBudgets.xhigh')" == "65536" ]] \
|
||||
&& _ok "with no base, host preferences come back (the frozen-file repair)" || _bad "no-base merge: $M4"
|
||||
[[ "$(printf '%s' "$M4" | jq -r .lastChangelogVersion)" == "1.4.0" ]] \
|
||||
&& _ok "and pi's own keys are kept" || _bad "no-base, pi key: $M4"
|
||||
printf '{oops' > "$M_IN/broken.json"
|
||||
jq -n --slurpfile T "$M_IN/broken.json" --slurpfile B /dev/null --slurpfile N "$M_IN/new.json" \
|
||||
-f "$MJQ" > /dev/null 2>&1 \
|
||||
&& _bad "unparseable input fails loudly for the caller to handle" || _ok "unparseable input fails loudly for the caller to handle"
|
||||
|
||||
t_name "privilege drop is the last thing that happens"
|
||||
tail -5 "$EP" | grep -q 'setpriv' && _ok "exec setpriv at the end" || _bad "setpriv last: $(tail -3 "$EP" | tr '\n' ' ')"
|
||||
grep -q -- '--bounding-set=-all' "$EP" && _ok "bounding set cleared" || _bad "bounding set cleared"
|
||||
|
||||
@@ -37,12 +37,15 @@ cat > "$AGENT/models.json" <<EOF
|
||||
EOF
|
||||
# sessionDir points at the *real* host home on purpose: that is the case the bridge
|
||||
# has to drop, and HOME must stay the real home for podman's sake.
|
||||
# defaultThinkingLevel is deliberately NOT xhigh here: the host has to disagree with the
|
||||
# ai-sandbox policy, otherwise "the policy wrote xhigh" cannot be told apart from "the
|
||||
# host said xhigh and we copied it".
|
||||
cat > "$AGENT/settings.json" <<EOF
|
||||
{
|
||||
"defaultProvider": "gateway",
|
||||
"defaultModel": "turbo",
|
||||
"theme": "dark/dark",
|
||||
"defaultThinkingLevel": "xhigh",
|
||||
"defaultThinkingLevel": "medium",
|
||||
"enableInstallTelemetry": false,
|
||||
"thinkingBudgets": { "xhigh": 65536 },
|
||||
"sessionDir": "$HOME/pi-sessions"
|
||||
@@ -110,13 +113,74 @@ t_name "settings travel, host paths do not"
|
||||
S="$(gen_file settings.json)"
|
||||
if [[ -n "$S" ]]; then
|
||||
for pair in '"defaultProvider": "gateway"' '"defaultModel": "turbo"' '"theme": "dark/dark"' \
|
||||
'"defaultThinkingLevel": "xhigh"' '"xhigh": 65536' '"enableInstallTelemetry": false'; do
|
||||
'"xhigh": 65536' '"enableInstallTelemetry": false'; do # the thinking level is policy, not a preference that travels
|
||||
grep -qF "$pair" "$S" && _ok "kept $pair" || _bad "kept $pair"
|
||||
done
|
||||
grep -qF 'sessionDir' "$S" && _bad "a value pointing at the host home is dropped" || _ok "a value pointing at the host home is dropped"
|
||||
fi
|
||||
assert_contains "dropped settings sessionDir" "and it says so"
|
||||
|
||||
# `--print-policy --keep` renders the bridge into the session directory and stops there:
|
||||
# a unit file never starts a container, and --keep keeps the rendered files to look at.
|
||||
rendered_level() { # rendered_level <level-or-default>
|
||||
clear_sessions
|
||||
sb --keep --print-policy --net none ${1:+--thinking "$1"}
|
||||
local f; f="$(gen_file settings.json)"
|
||||
[[ -n "$f" ]] || { printf 'none'; return; }
|
||||
jq -r '.defaultThinkingLevel // "absent"' "$f" 2>/dev/null || printf 'invalid'
|
||||
}
|
||||
|
||||
t_name "the thinking level is policy, written over the settings that travel"
|
||||
sb --keep --print-policy --net none
|
||||
assert_contains "thinking=xhigh (enforced on every run)" "xhigh is the default, not pi's medium"
|
||||
assert_contains "pi config: thinking -> xhigh" "and it is announced"
|
||||
# the fixture says medium, so an xhigh here can only have come from the policy
|
||||
assert_eq "xhigh" "$(rendered_level '')" "the policy beats the host file's medium"
|
||||
# the level is a preference, so it belongs to settings.json and nowhere else
|
||||
clear_sessions
|
||||
sb --keep --print-policy --net none
|
||||
if grep -q 'defaultThinkingLevel' "${M:-/nonexistent}" 2>/dev/null; then
|
||||
_bad "models.json is untouched by it" "the key appeared in the model catalog"
|
||||
else
|
||||
_ok "models.json is untouched by it"
|
||||
fi
|
||||
S="$(gen_file settings.json)"
|
||||
grep -qF '"defaultProvider": "gateway"' "${S:-/nonexistent}" \
|
||||
&& _ok "the rest of the settings still travel" || _bad "the rest of the settings still travel"
|
||||
grep -qF '"xhigh": 65536' "${S:-/nonexistent}" \
|
||||
&& _ok "including the host's thinking budgets" || _bad "the host's thinking budgets"
|
||||
grep -qF 'sessionDir' "${S:-/nonexistent}" \
|
||||
&& _bad "and a host path is still dropped while the level is added" || _ok "and a host path is still dropped while the level is added"
|
||||
|
||||
t_name "every level, including 'host' and 'off', and nothing in between"
|
||||
assert_eq "low" "$(rendered_level low)" "a level of your choosing"
|
||||
assert_eq "max" "$(rendered_level max)" "and the top of pi's range"
|
||||
assert_eq "off" "$(rendered_level off)" "off is pi's own level, not 'do not manage it'"
|
||||
assert_eq "medium" "$(rendered_level host)" "host leaves the key to settings.json (medium there)"
|
||||
rendered_level host >/dev/null
|
||||
assert_contains "thinking=host (not applied: the config decides)" "and says so"
|
||||
rendered_level bogus >/dev/null
|
||||
assert_rc 1 "an unknown level is refused"
|
||||
assert_contains "must be off, minimal, low, medium, high, xhigh, max or host" "with the list to pick from"
|
||||
sb_env AISB_THINKING=medium --print-policy --net none
|
||||
assert_contains "thinking=medium" "AISB_THINKING works"
|
||||
sb_env AISB_THINKING=medium --thinking max --print-policy --net none
|
||||
assert_contains "thinking=max" "the flag beats the environment"
|
||||
|
||||
t_name "a host with no settings.json still gets the policy"
|
||||
mv "$AGENT/settings.json" "$WORK/settings-away.json"
|
||||
assert_eq "xhigh" "$(rendered_level '')" "the level does not depend on the host having the file"
|
||||
clear_sessions
|
||||
sb --keep --print-policy --net none
|
||||
assert_contains "pi_endpoints" "the model bridge still happened alongside it"
|
||||
mv "$WORK/settings-away.json" "$AGENT/settings.json"
|
||||
|
||||
t_name "no bridge means no policy either, and it says so"
|
||||
clear_sessions
|
||||
sb --keep --no-pi-config --print-policy --net none
|
||||
assert_contains "thinking=xhigh (not applied: pi config bridging is off)" "--no-pi-config declines the thinking policy too"
|
||||
if [[ -n "$(gen_file settings.json)" ]]; then _bad "nothing is rendered for pi at all"; else _ok "nothing is rendered for pi at all"; fi
|
||||
|
||||
t_name "auth.json and the credential store are never bridged"
|
||||
if find "$AISB_STATE_DIR/sessions" \( -name auth.json -o -name models-store.json \) | grep -q .; then
|
||||
_bad "only models.json and settings.json are bridged"
|
||||
@@ -130,6 +194,7 @@ else
|
||||
fi
|
||||
|
||||
t_name "the token is passed by name, once, and announced"
|
||||
sb --keep --print-policy --net none
|
||||
assert_contains "passing \$AI_API_TOKEN from the host" "the auto-pass is announced"
|
||||
assert_eq "env_names=AI_API_TOKEN" "$(printf '%s' "$OUT" | grep '^env_names=' || true)" "exactly one variable is passed"
|
||||
assert_absent "$AI_API_TOKEN" "and only its name is reported"
|
||||
|
||||
Reference in New Issue
Block a user