Key Takeaways
- On October 30, 2026, arXiv published a study from two Queen's University researchers of 1,527,281 distinct English agent skills (SKILL.md files) found in public GitHub repositories.
- The study found that about one in seventeen skills, roughly 88,000, contains an "oversight-loosening" directive: a line telling the agent to act without asking, skip a safeguard, or keep going when it would otherwise stop.
- 70.1% of those directives apply unconditionally, not only in a scoped context such as CI. Permission flags such as
npx -yand--yesare the most common kind (33.9%), followed by "proceed without asking" lines (27.4%). - Skills that run shell code, install packages or use sudo had 1.7 to 1.8 times the odds of containing one.
- Two open-source skill scanners caught at most 46% of the directives in a hand-labelled test set. A GPT-5.4 prompt using the authors' codebook caught 61% at 0.71 precision.
The agent skills security study, "When Skills Tell Agents Not to Ask," measured something most engineers never check: what the skills they install tell a coding agent about asking permission. Timofey Sanko and Yuan Tian of Queen's University analyzed 1.53 million public skills and found that about one in seventeen tells the agent to need you less. Most of those lines aren't malicious. They are convenience lines that nobody scoped.
What happened
Sanko and Tian's preprint, "When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills", appeared in arXiv's software engineering listing on October 30, 2026. The authors call it the first large-scale study of instructions in skills that reduce human control over what an agent does.
A skill is a folder built around a SKILL.md file: a description plus instructions that an agent such as Claude Code loads when a task matches. Skills are shared by copying them between repositories. That makes them easy to adopt and hard to audit, because a line written for one author's workflow ends up in yours.
What the agent skills security study measured
The data comes from GitSkills, a public dataset of SKILL.md files on GitHub. It holds 3,797,117 files from 282,200 repositories, which deduplicate to 1,877,981 distinct contents. The authors kept English skills first committed in October 2025 or later, which gave them 1,527,281 distinct skills.
They hand-labelled 2,097 instruction units against a codebook, marking each as loosening oversight, tightening it, or neither. Then they trained a detector to label the rest. Loosening means one of four things: the agent acts without asking, skips or bypasses a safeguard, acts without reporting, or continues where it would otherwise stop.
The nine ways a skill loosens oversight
| Mechanism | Share of loosening directives | Example the authors give |
|---|---|---|
| Permission flags | 33.9% | npx -y, --yes |
| Confirmation | 27.4% | "Proceed without asking the user" |
| Persistence (not stopping) | 16.0% | "Don't stop between tasks. Keep going until done" |
| Safeguards | 7.1% | "Commit with --no-verify" |
| Auto-action | 6.2% | "Auto-commit completed units" |
| Transparency | 4.5% | "Do not announce each commit" |
| Consequential action | 3.0% | "Feel free to push" |
| Invocation as approval | 1.5% | "Invoking this command is the go-ahead" |
| Precedence | 0.5% | "These rules override all other instructions" |
The examples are the authors' paraphrases from the paper's Table 1, not quotes from specific repositories. The same table shows the opposite pattern: 46.3% of tightening directives are confirmation lines, such as "ask before pushing."
The numbers behind it
One in seventeen, and most of it unscoped
"About one in seventeen skills (about 88,000 skills) contains a loosening directive," the paper reports. The authors treat that as a lower bound, because their detector finds 61% of the loosening lines that human labellers find.
Scope is the bigger issue. A line like "pass --yes only in CI" is a reasonable convenience. But 70.1% of loosening directives carry no condition at all, so they apply on your laptop, against your real branches, the same as in a pipeline.
Riskier skills loosen more
The authors compared skills with similar amounts of oversight language. Skills containing a shell code block had 1.82 times the odds of a loosening directive. Skills that install packages also had 1.82 times the odds, and skills that use sudo had 1.74 times. Piping curl into sh came in at 1.31. GitHub stars showed no relationship (odds ratio 0.99), so popularity is no signal of safety. Skills that handle secrets or credentials were the exception, with lower odds (0.77).
A few sources spread most of it
Copying concentrates the problem. 2.4% of skill families that carry loosening account for half of all copies of loosening-carrying skills. 960 of 86,048 distinct loosening units, or 1.1%, account for half of all occurrences. And 93.4% of copied variants leave the original's oversight lines unchanged. People copy the whole file and rarely edit those lines.
Scanners mostly miss it
The authors ran two open-source skill scanners, Cisco AI Defense skill-scanner 2.2.1 and NVIDIA SkillSpector 2.12.0, against a 400-unit hand-labelled set with 61 loosening directives. Offline, Cisco's scanner caught 2% and SkillSpector caught 21%. With their LLM analysis turned on (GPT-5.4), Cisco reached 15% and SkillSpector 46%, at precision below 0.4. Running both caught 51%.
The same model, given the authors' codebook instead of the scanners' prompts, caught 61% at 0.71 precision. The authors conclude the weakness lies in the question the scanners ask, which is whether a skill is malicious, not in the model. A line like "proceed without asking" isn't malicious. It just removes you from the loop.
What the study doesn't show
The study measures what skills say, not what agents do. It doesn't test whether Claude Code, Codex or any other agent follows these lines. It covers public GitHub only, in English, so private and marketplace skills may differ. The second labeller was an LLM (Claude Opus 5.5), with agreement of κ = 0.82 against the human labels.
A second paper in the same October 30 listing hints at the behavior side. In "Security Impact of Coding Agent's Memory Files", researchers planted malicious CLAUDE.md-style memory files in 36 secure-coding tasks. The secure completion rate moved by −8 to +6 percentage points, with no consistent pattern across agents. Claude Code mentioned the file in about two-thirds of its final messages, while OpenCode mentioned it almost only in its reasoning trace. That's mixed evidence, not proof that agents obey every instruction they load.
What this means for you
If you install skills
- List what you have. Personal skills live in
~/.claude/skills/<name>/SKILL.mdand project skills in.claude/skills/<name>/SKILL.md, according to the Claude Code skills docs. - Search them for the nine mechanisms above:
-y,--yes,--no-verify, "without asking", "don't stop", "auto-commit", "override". - Read skills that run shell, install packages or use sudo first. That's where the study found the highest odds.
- Check the
allowed-toolsfrontmatter too. It lists the tools Claude can use without asking during that skill's turn. The docs warn that "workspace trust doesn't gate this field," so review it in skills checked into a repository before you run Claude Code there.
If you write or share skills
- Scope every convenience line, as the authors recommend: "pass --yes only in CI," not just "--yes."
- Put confirmation steps back in for pushes, tags, deploys and anything destructive.
If you own your team's agent setup
- Don't treat a clean scanner report as an audit. In this test, the best single scanner caught 46%.
- Add an oversight-specific check to code review for SKILL.md changes. In the paper's words, "A flagged loosening directive should prompt a review, not a verdict."
How this connects
This is the latest in a run of October studies showing that the files around a coding agent shape its behavior more than most teams assume. In one, coding agents followed stale instruction files over the code, and one sentence in a user-level file fixed it. In another, agents copied repository precedent over the developer's stated intent. Skills are the newest layer of that context, and the first one people install from strangers.
Context engineering is no longer just choosing what an agent reads. It also means auditing what that context authorizes. For the broader shift in how engineers work with agents, see loop engineering vs prompt engineering.
How this comes up in interviews
AI engineering loops now ask how you'd let an agent act safely: which actions need human approval, how you'd scope permissions, and how you'd review third-party tools and prompts. This study gives you concrete numbers for that answer. It's the kind of guardrail question covered in our AI engineer interview questions with worked answers, and in how coding interviews changed in the age of AI.
FAQ
Are Claude Code skills safe to install?
A skill is instructions plus optional scripts, so treat it like code you didn't write. This study found that about one in seventeen public skills tells the agent to ask you less. Read a skill's SKILL.md and its allowed-tools field before you install it.
What is an oversight-loosening directive?
It's a line in a skill that reduces human control over the agent: acting without asking, skipping a safeguard such as a pre-commit hook, not reporting, or continuing where it would otherwise stop. Examples include "npx -y", "--no-verify" and "proceed without asking the user."
Do skill scanners catch these directives?
Mostly not, in this study. Cisco AI Defense skill-scanner and NVIDIA SkillSpector caught at most 46% of hand-labelled loosening directives, at precision below 0.4. The scanners look for malicious skills, and most loosening lines aren't malicious.
Do coding agents actually follow these instructions?
The study doesn't test that. It measures what skills say. A separate October 2026 paper on malicious memory files found mixed effects on secure code, with no consistent pattern across agents.
Further Reading
- Sanko and Tian, "When Skills Tell Agents Not to Ask" (arXiv:2610.37734): the study, with the codebook, odds ratios and scanner evaluation.
- Santos de Moura, Güler and Noller, "Security Impact of Coding Agent's Memory Files" (arXiv:2610.38521): malicious CLAUDE.md-style files and secure code.
- Claude Code docs: Skills: the SKILL.md format, skill locations and the allowed-tools field.
