Your Agent's Config Is a Dependency With No Lockfile
Your dependency policy is probably decent. Locked versions, hash verification, an SBOM if you're serious, a review gate for new packages.
Meanwhile your coding agent imports instructions, skills, hooks, tool declarations, and subagents from public marketplaces and repositories. It runs all of it with your privileges, in your working directory, continuously. And none of it is in the lockfile, because the harness, as researchers now call this layer, has no lockfile.
What's Actually in the Harness
Inventory it sometime: the instruction files, the skills directory, the hooks that fire on events, the MCP server declarations, the subagent definitions. Each of these is code-shaped. Each was probably installed by copying a snippet from a README or a marketplace listing, sometimes written by a stranger, sometimes generated by another model.
We spent a decade teaching teams to treat npm installs as untrusted input. Then we built a new dependency layer, made it executable, made it ambient, and gave it a filename that sounds like configuration.
Someone Finally Measured It
In September 2026, researchers published the first systematic scan of the harness: 3,171 public GitHub repositories with assembled agent setups, 2,660 of them combining multiple component types, plus 511 published skill collections.
The results, and the method deserves respect: every finding was independently re-derived from the repository at its pinned commit, disagreements went to an adjudicator, and a second model session re-checked every counted pair.
16% of setups carry a security defect. The breakdown:
- 9.8% install an MCP server with no version pinned. What you reviewed is not necessarily what runs.
-
3.1% pre-approve arbitrary execution behind a scoped-looking grant. A permission like
Bash(python:*)reads as contained and means anything Python can do, which is anything. - A third class ships inside skill collections: skills that pre-approve the shell for whoever installs them, arriving through marketplaces where a scan could theoretically see them.
The scary reading is the exploit chain. The accurate reading is quieter and worse: these are defects in ordinary setups, assembled by ordinary developers, following ordinary READMEs. Nobody got hacked in the study. Everybody got exposed.
Why This Is Not Like Packages
A dependency executes at build or test time, in a context you scaffolded. The harness runs as you, all the time, in the directory where your work and your secrets live. And its permissions read like configuration to a human scanning a PR diff, while functioning like a grant of execution to a machine reading them literally.
The scoped-looking grant is the perfect emblem: a permission string designed to feel surgical, meaning "run anything."
The Hygiene, Until the Ecosystem Grows Up
Pin everything. MCP server versions, skill sources, hook scripts. What changed since install is the first question an incident will ask.
Review config like dependency changes. The .claude directory and its cousins go through PRs, not direct commits. A config diff is a supply-chain diff.
Scan it. The study's instrument, corpus, and verdicts are released. Run it against your own setups before someone else's paper does.
Grant minimally, out loud. Enumerate what is pre-approved in your agent config, in a file a human reads monthly. The pre-approvals you forgot are the ones that get you.
💡 Pro-tip: the one-line audit: list every permission in your agent config that lets something execute without asking. If that list is longer than you remembered, you have found the article's point in your own repository.
What the Scan Misses
Scanning catches rule-decidable defects only. Intent cannot be scanned, and no lockfile standard exists for this layer yet, so this is manual discipline for now: pins, reviews, scans, minimal grants.
The harness is still worth it. So was npm. We just know how the first chapter of this story goes.
What's in your agent config directory that you didn't put there, and when did you last look?
References
- Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations (September 2026), instrument and data released: arxiv.org/abs/2609.07360
Top comments (1)
The lockfile framing lands, and "code-shaped but named like configuration" is the line I'd keep. There's a second gap it opens, though: pinning tells you what was installed, not what a particular run actually read.
A pinned MCP version, a reviewed config diff and an enumerated permission list all describe the setup at deploy time. During a run the same setup still reads ambient values — a clock, a feature flag, a database row, the model and tool revisions actually served, the routing choice made upstream. A deterministic rule that consults any of those is reading an input it never declared. Two runs with byte-identical config can take different branches, and the lockfile can't tell you which read differed. That's the same boundary as your own caveat: a scan catches rule-decidable defects, but "was this rule evaluated on this input, against these values" is a per-run fact, not a property of the config.
Which is why I'd pair the lockfile with a run record: the config revision, plus the external values the node read and the version of each tool that answered. Without it, replaying against pinned config is still a guess about ambient state, and a regression and an environment change read the same way.
Do you record the values a run read alongside the pinned config, or leave replay to reconstruct them?
Treating skills, hooks, and MCP servers like ordinary config is how unpinned copies slip in. I would inventory that layer the same way I inventory packages: pin the version, record a hash, and refuse a new source until someone has reviewed what it can run.
Honestly, most of it I didn't put there on purpose, which is the point of the article. A simple habit is to open the config folder once a month and read every instruction file top to bottom, deleting what you can't explain. Putting the folder under version control makes later changes visible, so a surprise shows up as a diff instead of as odd behavior.
The 3.1% is what happens when the grant lives in a file the agent's own runtime interprets:
Bash(python:*)reads as scoped to the reviewer and as everything to the machine.In the sandbox I run, the grant sits on the other side of the door. The MCP server's
policy.jsonnames the tools each identity may use, a tool that isn't granted isn't listed, and a call to it gets the same answer as a tool that doesn't exist. The destructivedelete_recordis granted to no one, and a static check fails the build if that ever changes.A monthly read finds a bad grant after the fact, where a static check refuses it at build time. That is the lockfile for permissions.