AI Coding Agent Sandbox: Fixing Approval Fatigue in CI

My test for any agent permission design is what happens at 3 a.m. on a CI runner when nobody is there to click Allow. Either the job hangs on a prompt, or someone adds a flag that turns every prompt off, and from that moment the only thing between the agent and your secrets is whatever boundary the runner already had.

Anthropic published a long engineering write-up on how it contains Claude across its products, with numbers from its own telemetry and one uncomfortable red-team result. I read it next to the Claude Code sandboxing, sandbox environments and auto mode documentation and the open-source sandbox runtime README. Every figure below comes from those pages; I ran none of this myself, and where a number is small or self-reported I say so. This is the containment angle, separate from my earlier posts on the permission layers and the auto mode classifier.

Approval prompts measure attention, not safety

The number that reframes the whole write-up is one line of telemetry: users approved roughly 93% of Claude Code permission prompts. A control that is accepted nine times in ten is not filtering anything the person is still reading. Anthropic adds that experienced users auto-approve roughly twice as often as new users, which is what habit looks like in data.

Anthropic frames agent risk as likelihood times blast radius. Training, classifiers and prompts push likelihood down, but every new capability and every new credential pushes blast radius up. It then separates two ways to respond: supervise what the agent does, or contain what it can do. The article is about the second, and its principle is to design containment at the environment layer first and steer behaviour at the model layer afterwards.

In CI the argument is sharper, because there is no human to be fatigued. The Claude Code documentation says to run sessions started with the skip-permissions flag only inside a container, a VM or the sandbox runtime, and notes that Claude Code refuses to start with that flag as root on Linux and macOS. So the real decision for a pipeline is never prompt or no prompt. It is where the boundary sits.

The default boundary: reads open, writes scoped, network shut

Anthropic describes the original Claude Code design as allowing reads and requiring approval for writes, bash and network access. It then added an OS-level sandbox, Seatbelt on macOS and bubblewrap on Linux, where reads are allowed, writes are allowed inside the workspace and network is denied by default. The result was an 84% reduction in permission prompts, and the runtime was open-sourced so the boundary is auditable. The documentation spells out exactly what a sandboxed command gets.

Read the second row twice. Out of the box the sandbox stops a command writing outside the project and phoning home, but it does not stop it reading your SSH keys or cloud credentials, and it hands over your environment variables. The documentation is blunt that there is no built-in credential deny list, so only the files and variables you list are restricted. That gap is exactly where the red-team result in the next section lands.

The red-team prompt that model-layer defences cannot see

In February 2026, during an internal exercise, a researcher phished an employee into running a prompt that asked Claude to read ~/.aws/credentials, encode it and POST it to an external server. Across 25 retries Claude completed the exfiltration 24 times. Anthropic's reading is that the user typed the instruction, so model-layer defences had nothing to catch. Only egress controls and filesystem boundaries would have stopped it.

The contrast is worth sitting with. On Gray Swan's Agent Red Teaming benchmark Anthropic reports Claude Opus 4.7 holding attack success to roughly 0.1% on single attempts and around 5 to 6% after 100 adaptive attempts, which is a strong result about injected content. It says nothing about an instruction that arrives from the legitimate user's own keyboard, because training shapes tendencies, not capabilities. A hostile pasted prompt is a capability question, and only the environment answers capability questions.

The configuration below closes that specific path using keys from the Claude Code documentation. File entries with mode deny are blocked for reads inside the sandbox, environment variables are unset before each sandboxed command, and the empty allowlist leaves no route out. Then verify it rather than trust it: ask Claude to run both probe lines, because commands you type yourself at the shell-mode prompt usually run outside the sandbox and prove nothing.

{
  "sandbox": {
    "enabled": true,
    "credentials": {
      "files": [
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.ssh", "mode": "deny" }
      ],
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" }
      ]
    }
  }
}

# Ask Claude to run each line. Typing them at the ! prompt tests nothing,
# because shell-mode commands usually run outside the sandbox.

touch ~/sandbox-probe
# Sandboxed: "Operation not permitted" on macOS, "Read-only file system" on Linux/WSL2.
# If it succeeds and home is not a writable sandbox path, the sandbox is off: delete the file.

curl --noproxy '*' https://example.com
# Sandboxed: "Could not resolve host", because there is no route around the proxy.

An allowlist is a capability grant, not a destination filter

Anthropic describes a third-party disclosure in which a malicious file in the workspace carried hidden instructions and an attacker's API key. Claude used the Files API on api.anthropic.com, a host the allowlist permitted, to upload files to the attacker's account. Anthropic writes that it had previously conceptualised the allowlist as a destination filter. The reframing is that each allowed host is a set of capabilities, including whatever an attacker can do with an account on that host.

The fix in Anthropic's VM product was a proxy inside the VM that only passes requests carrying the VM's own session token and blocks server-side fetch headers. The Claude Code documentation reaches the same conclusion for the built-in sandbox. Because the proxy decides from the client-supplied hostname and does not inspect TLS by default, broad entries such as github.com can create exfiltration paths and code inside the sandbox may use domain fronting. If your threat model needs more, the documented route is a custom proxy that terminates TLS, and the docs call stronger TLS-aware isolation an active area of development.

In practice, for each hostname you add, ask what a hostile process could do with its own account there. A package registry you only read from is a different grant from a code host or an LLM API you can write to. One detail helps on cloud runners: after a hostname passes the allowlist the proxy resolves it and refuses names that resolve only to local addresses, and the documentation names the 169.254.169.254 cloud metadata endpoint among them.

Auto mode belongs inside the sandbox, not instead of it

The containment article says Claude Code auto mode catches roughly 83% of overeager behaviours before they execute while blocking roughly 0.4% of benign commands, and calls it one layer inside a sandbox rather than a substitute. The auto mode post gives the table below. Its own wording is that the 17% miss rate on real overeager actions is the honest number, and that set holds only 52 actions, so treat it as a small sample.

The design has two properties that matter for unattended runs. The classifier sees only user messages and tool calls, with assistant text and tool results stripped, so the agent cannot argue its way past it and injected content has less to work with. And a denied action returns to the agent with a reason, but three consecutive denials or 20 in total escalate to the human, and in headless mode the process terminates. In CI, a job that dies on repeated denials is the correct failure.

The two controls compose without overlapping. The sandbox documentation says sandbox auto-allow approves a Bash command because the boundary contains it, while auto mode uses a classifier to review the action, and the two can be combined. The sandbox environments page adds that a classifier is a per-action control, not an isolation boundary, so a boundary still adds defence in depth for auto mode. For the skip-permissions flag it is not optional. Anthropic's own caveat is that auto mode is not a drop-in replacement for careful human review on high-stakes infrastructure.

Picking the boundary for a CI runner

The built-in sandbox wraps shell commands only. The sandbox environments page says that on its own it is not sufficient for fully unattended runs in either skip-permissions or auto mode, and lists the options by what each one isolates.

The documentation calls a custom container the most common path for organisations with existing container infrastructure or CI runners, and a VM the right choice for untrusted code. If you stay with the sandbox runtime, put the config below in a file and pass it with the settings flag. The domain and write paths follow the documented minimum for a Claude Code session on Linux: your project, the two Claude config paths, the temp directory and the API host. Unprivileged containers need the weaker nested sandbox mode, which the documentation says considerably weakens security.

# ci-srt.json: the sandbox runtime config for a Linux CI runner
{
  "network": {
    "allowedDomains": ["api.anthropic.com", "registry.npmjs.org"],
    "deniedDomains": []
  },
  "filesystem": {
    "denyRead": ["~/.ssh", "~/.aws"],
    "allowWrite": [".", "~/.claude", "~/.claude.json", "/tmp"],
    "denyWrite": [".env", ".claude/settings.json", ".claude/settings.local.json"]
  }
}

# The runtime only grants writes to paths that already exist on Linux,
# so create Claude Code's config paths before the first launch.
mkdir -p ~/.claude && { [ -f ~/.claude.json ] || echo '{}' > ~/.claude.json; }

# Wrap the whole Claude Code process, not just its shell commands.
npx @anthropic-ai/sandbox-runtime --settings ./ci-srt.json claude

Three traps from the runtime documentation. If the settings file does not exist and you pass no flag, the runtime still starts, with network blocked and writes confined to built-in paths, so a clean start proves nothing about your settings loading. On Linux the deny list is built once at launch, so anything the session creates later, such as a fresh git clone, is not covered. And your write grants still include paths Claude Code loads configuration from, so deny them with denyWrite or a session can persist hooks that run unsandboxed next time. The runtime itself is labelled a beta research preview whose config format may change.

Where the boundary does not reach, and where it fails open

The sandbox has a precise edge, and most real incidents live just outside it. The documentation lists what runs outside it:

• Read, Edit, Write, WebFetch and WebSearch follow permission rules instead, so a denyRead entry does not stop the Read tool and allowedDomains does not limit WebFetch.
• Command hooks, local MCP servers, LSP servers, plugin monitors and helpers such as apiKeyHelper run with your full access.
• If the sandbox cannot start because a dependency is missing, Claude Code runs commands unsandboxed unless failIfUnavailable is true.
• Claude can ask to retry a failed command outside the sandbox, and any command matching excludedCommands runs with no filesystem limits and no proxy.

Anthropic's write-up also records three reported vulnerabilities between mid-2025 and January 2026 in which code ran before the user accepted the trust dialog, for example a committed settings file hook that ran on startup. The fix was to defer parsing project-local config until after trust. For a runner that checks out pull request branches, repository-controlled config is attacker-controlled input. The documentation's answer is an admin-required sandbox, where Claude Code ignores a repository's settings that loosen it, and the first block below shows how narrow an exclusion has to be.

// Wrong: covers everything the docker CLI can do, outside the sandbox.
{ "sandbox": { "enabled": true, "excludedCommands": ["docker *"] } }

// Right: the narrowest pattern that still lets the one tool you trust run.
// Patterns match the text of the call, so a script that calls docker
// internally does not match, and neither does /usr/local/bin/docker.
{ "sandbox": { "enabled": true, "excludedCommands": ["docker compose *"] } }

Anthropic's closing lesson fits this section: standard primitives held while custom code failed. The component that broke in its major incident was the custom proxy in claude.ai code execution, not gVisor. The trade-off I would name is that the 84% prompt reduction is not free. The unsandboxed retry reintroduces the exact prompt you removed, so setting allowUnsandboxedCommands to false trades some convenience for a boundary the model cannot negotiate with.

What to do on Monday

Treat this as a checklist for any place an agent runs without a person watching. The 93% and 84% figures are Anthropic's telemetry on its own users, not a benchmark of your repository, so measure your own prompt counts before and after.

1. List every place an agent runs unattended, and write down what each can read, write and reach. If the answer for network is anything, that is the first fix.
2. Match the boundary to the mode: the skip-permissions flag or auto mode in CI goes inside a container, VM or the sandbox runtime, and developer laptops use the sandbox panel.
3. Deliver managed settings with enabled, failIfUnavailable and allowUnsandboxedCommands set, plus credentials entries for the secrets your runners hold.
4. Start allowedDomains empty, add hosts one at a time, and turn on strictAllowlist so a blocked host fails the job instead of prompting.
5. Run the two probes, then borrow Anthropic's canary idea: plant a fake credential string and alert if it ever appears in egress logs.

{
  "sandbox": {
    "enabled": true,
    "failIfUnavailable": true,
    "allowUnsandboxedCommands": false,
    "network": {
      "strictAllowlist": true,
      "allowedDomains": ["registry.npmjs.org"]
    },
    "credentials": {
      "files": [
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.ssh", "mode": "deny" }
      ],
      "envVars": [{ "name": "NPM_TOKEN", "mode": "deny" }]
    }
  }
}

# Linux runner preflight
sudo apt-get install bubblewrap socat

# Ubuntu 24.04+: 1 means bubblewrap needs the AppArmor bwrap profile.
# 0, or "No such file or directory", means skip it.
sysctl kernel.apparmor_restrict_unprivileged_userns

# One-off strict session with no unsandboxed retry:
claude --settings '{"sandbox": {"enabled": true, "allowUnsandboxedCommands": false}}'

# Catch misconfigured domain entries (ambiguous IPv6 spellings, no TLS termination):
claude doctor

The managed settings block uses keys the documentation lists for enforcement and the strict allowlist, which needs Claude Code v2.1.219 or later. The shell block is the preflight I would run on a new Linux runner before trusting any of it.

The rule I take from this: a prompt is a request for attention, and attention does not scale to a pipeline, so move the decision into a boundary the model cannot negotiate with, then use classifiers and prompts for what the boundary cannot express. Put credentials out of reach and the network shut before you turn any approval off.