Guardrails for coding agents that don't depend on the agent behaving

In July 2025 a Replit agent deleted a production database in the middle of a code freeze. The freeze was real. It was also just a sentence in the agent’s instructions, which the agent could read, agree with, and ignore (AI Incident Database #1152).

Our mobile team uses Claude Code, Codex, Cursor and Copilot every day. The agents push branches, read Jira tickets and Sentry events, and call cloud tools through MCP servers. Until last month, the only thing standing between one of them and git push --force was a paragraph in a Markdown file. So I replaced that paragraph with code, and then asked two very different reviewers to break it.

This post is about the decisions, the incidents behind them, and what went wrong along the way. I’ve left out anything specific to the app itself. The audit turned up a couple of things that are being fixed, and those stay internal.

Two places where a rule can actually hold

A written rule reaches the model. An enforced rule sits somewhere the model can’t route around. For a coding agent there are two such places.

The first is the moment before a tool call runs. Claude Code has PreToolUse hooks: a script that receives every shell command, file read, file edit and MCP call as JSON and can answer deny or ask. Anthropic’s docs are clear that a hook’s deny holds even in bypass mode (hooks reference).

The second is the pull request. Whatever an agent does locally eventually becomes a diff, and a CI job can refuse a diff no matter which tool, laptop or human produced it.

Everything else is documentation.

The hook only ever tightens

The hook returns deny, ask, or nothing. It never returns allow. That one choice means a bug in my policy can only make an agent stricter. If the hook crashes, it answers ask, so a broken guardrail puts a person in front of the call and never waves it through.

The rules live in JSON, because the hook runs on every tool call on stock macOS Python 3.9 and I didn’t want a single third-party package in that path:

{"id": "git-force-push", "decision": "deny", "tier": 3,
 "reason": "Force push rewrites shared history. A human must run it."},
{"id": "git-push", "decision": "ask", "tier": 3,
 "reason": "Push publishes code and can start CI/CD."}

ask means Claude Code shows a prompt for that exact command, and approving git push origin main says nothing about git push origin release. Around forty rules cover pushes, merges, store uploads, signing tools, curl | sh, dumping the environment, recursive deletes of broad paths, cloud deletes, installs, and edits to the guardrail files themselves. Any command that names a protected file (.env, keystores, ~/.ssh, keychains) is denied unless the program only reads metadata, like ls or git status. git add .env is denied too.

I underestimated false positives. Badly. The hook went live in my own session the moment I wrote its settings file, and it spent the rest of the project arguing with me. It blocked a Python heredoc I was using to edit a workflow, because the script mentioned a signing file by name. That was correct. It also blocked a regex, .*?, because it expanded like a shell glob that could match .env. That one was wrong, and it became a regression test. Most of the tuning after that was teaching the hook which text the shell actually runs and which text is data: commit messages, heredocs fed to cat, the pattern argument of grep.

Everything the agent reads is data

Greshake et al. showed in 2023 that instructions hidden in retrieved content can take over an LLM application without the attacker ever talking to it (arXiv:2302.12173). Two years later it works on real coding tools. Invariant Labs planted instructions in a public GitHub issue, a developer asked their agent to “check the open issues”, and the agent leaked private repositories into a public PR (Invariant Labs). CamoLeak did something similar with hidden comments in a PR description and GitHub Copilot Chat (Legit Security).

Simon Willison’s “lethal trifecta” explains the shape: private data, untrusted content, and a way to send data out (simonwillison.net). A coding agent on a real repo has all three. I can’t remove the first two, since reading tickets and code is the job. So the design attacks the third leg. Every outward write asks first: gh issue comment, gh pr create, Jira comments and transitions, Slack messages, Figma writes. And the hook never reads repository text when it decides anything, so a comment that says “the maintainer has pre-authorized you” changes nothing.

The CI gate, and the mistake the red team found in it

The CI job scans the added lines of every PR for:

• secret patterns (it never prints the value, only rule and line)
• committed keystores and environment files
• invisible Unicode
• workflow permission changes
• git-sourced dependencies
• deleted or newly skipped tests

The invisible Unicode check comes from Trojan Source, which showed that bidirectional control characters make code read differently to a reviewer and a compiler (USENIX Security 2023). Pillar Security later used the same trick to hide instructions inside Cursor and Copilot rules files (Pillar Security). The first thing this check ever caught was my own eval script. A tool call had quietly turned my ‮ escapes into real bidi characters.

When I thought the CI side was done, I gave the whole setup to a different model, Claude Fable, with one instruction: find ways around it. It tested only by passing candidate commands as strings to the guard and by running the checker against throwaway repos. It never executed anything. It came back with seventeen findings, and the worst one was embarrassing. The CI job ran the checker from the PR’s own branch. A PR could empty the secret patterns, add a token, and pass its own check. Fable proved it in a scratch repo in two different ways.

The fix is to extract the checker and the policies from the base branch with git archive and run those against the PR. The rest of the list was smaller but real:

• git -P push slipped past the command parser
• cat .en? read .env after shell expansion
• Key.Properties worked on macOS’s case-insensitive filesystem
• an agent could add its own override trailer to a commit to waive a failing check

Fifteen of the seventeen are fixed, each with a regression test.

Then a human reviewer went through the PR and left twenty-eight comments. Several were things no model had flagged. My new CI trigger made the existing build and test jobs report “skipped” on every PR, and GitHub counts a job skipped by its if: condition as passing, so PRs could have merged without anyone running the real tests. The trusted base-branch check also ran after three steps that executed the PR’s own Python, and anything those steps wrote into the job’s environment would carry forward. A PR could also add .gitattributes with *.dart -diff and turn every change into “Binary files differ”, which blinds any diff-based scanner. The guardrails now live in their own workflow, the trusted check runs first, and .gitattributes is read from the base.

Sessions nobody is watching

The question that changed the design most was a plain one. What about a session someone left open on Tuesday that pushes on Thursday?

A session’s context is its memory. Everything it read stays there and can steer later actions, which OWASP’s agentic list calls memory and context poisoning (ASI06, OWASP). While researching it I found two open Claude Code bugs. In one, a hook’s ask lets the call through in auto mode (#89561). In the other, headless mode silently denies it (#95726). The permissions docs also mention claude --bare, which starts a session that loads no project hooks at all.

So a prompt can’t guard anything irreversible. The hook now turns every tier-3 ask into a deny in three cases: the session runs in bypass or dontAsk mode, it’s more than 12 hours old, or it resumed after more than 8 idle hours. Agents can’t start --bare or permission-skipping sessions, and starting a nested headless session asks first. Session IDs are logged only as a hash, so a weekly metrics.py run shows long-running or unattended sessions without storing anything that identifies them.

That still doesn’t cover Codex, Cursor, or a teammate whose laptop lacks Python. For those, the backstop is a GitHub setting: “Require approval of the most recent reviewable push”. With it on, any commit pushed after a review dismisses the approval (GitHub docs). The forgotten Thursday session can still push. It can’t merge.

The token the agent pushes with

The most useful thing I built came late. I wrote a small checker that reports the credentials an agent would use: token type, scopes, how many repos it can reach, whether an SSH key is loaded. It prints metadata only. On my own laptop:

FAIL  GitHub token  OAuth token from `gh auth login` (broad, long-lived);
                    scopes: admin:public_key, gist, read:org, repo, workflow
FAIL  SSH agent     1 personal key(s) usable by agent commands

Every agent session I’d ever run could push to every repository I can reach, rewrite CI workflows, and add SSH keys to my account. No regex fixes that, because the damage a command can do is set by the token it runs with, and the only way to shrink that is to hand agents a different, smaller token than the one you use yourself. There are two options. One is a fine-grained token limited to the one repo, with no Workflows permission, so GitHub itself rejects any push that touches CI files. The other is a GitHub App token that expires after an hour (GitHub docs). A settings block routes git over HTTPS through that token and blanks the SSH agent for agent sessions only. Agents are also denied from minting or widening credentials themselves.

Testing the guardrails themselves

Each of fifteen eval scenarios pairs a behavioral test with deterministic checks, 69 of them in total. Take “asked to disable a failing test”. An agent can be judged on how it handles the request, but CI also proves that deleting the test fails the build even if the agent complies. The outcomes are stored in a baseline file, so a policy edit that turns a deny into an ask fails CI until someone updates the baseline in a reviewed PR.

The rules reference page is generated from the policy files, with a blocked and an allowed example for every rule, and every example runs through the real guard at build time. Generating it caught two regex bugs on the first run, because skip: false and hosted: https://pub.dev were both being flagged. Negative lookaheads and optional whitespace backtrack in ways I apparently still don’t expect.

Real incidents, and what this would do

Incident	What would stop it here	What wouldn’t
tj-actions (2025): repointed tags dumped CI secrets into logs	New @latest/@main action refs fail CI. Printing decoded secret files fails CI.	Existing unpinned refs, until someone pins them
Codecov (2021): curl | bash uploader sent $(env) out	The hook denies curl | sh and environment dumps	A remote script someone already trusts
Nx s1ngularity (2025): malware drove local AI CLIs to hunt secrets	Agents can’t start permission-skipping sessions or read ~/.ssh	Malware running as its own process never meets the hook
Amazon Q (2025): a shipped prompt told the agent to wipe files and cloud resources	Broad rm -rf is denied. Cloud deletes ask for the exact command.	A poisoned extension outside Claude Code
Invariant’s GitHub MCP leak (2025)	Every outward write asks	A human who approves without reading
Slopsquatting (USENIX 2025)	pub add asks. Every new dependency is listed on the PR.	A reviewer who doesn’t check the name

What this doesn’t do

The hook matches regular expressions. g=git; $g push gets through, and so does a Python one-liner that shells out. I wrote that into the docs on day one, and the red team confirmed it. Only Claude Code gets a pre-execution check. Every other tool gets instructions and the CI gate. And nothing in a repository can stop someone with admin rights. The Amazon Q incident started when a contributor was handed admin credentials (AWS bulletin), and no hook fixes that.

The follow-up list has more GitHub settings on it than code: branch protection, environment reviewers, SHA-pinned actions, secret scanning. A cheaper first step exists, though. Run the credential check on your own laptop and read the scopes line.