# Your AGENTS.md is the highest-leverage code you're not testing

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/blog/agents-md-testing
Source observed: 2026-11-15T21:02:53.899Z
Last checked: 2026-11-16T18:25:19.929+00:00
Indexed: 2026-11-15T21:04:26.893+00:00

---

# Your `AGENTS.md` is the highest-leverage code you're not testing

**April 8, 2026 · 3 min read**  
By [Ben, Founder, Stet](https://www.stet.sh/about)

I shipped a bad `AGENTS.md` change. The agent became less aligned and started quietly over-engineering changes: instead of writing a clean, 20-line function, it wrote a bloated 40 lines. That change degraded the quality of every engineer using coding agents in our monorepo with 100 developers.

## The measurement gap

We did not stop the change because we did not know anything was broken. A recurring pattern is treating `AGENTS.md` like generated code: fire and forget, hope it does not break anything, and tweak it whenever the model changes.

`AGENTS.md` is a high-leverage point in the harness, pulled into every turn for every engineer in the codebase. Every line deserves careful consideration. If agents write **80 to 100 percent** of your code, this fundamental harness component deserves the same craft as important infrastructure.

## Why this is hard

You cannot diff the output of an `AGENTS.md` change. Agents are non-deterministic by necessity, and one instruction line may not produce a reliable, consistent behavior change. The same edit can produce different results across tasks, repositories, models, harnesses, and time.

The feedback loop is difficult to close. An organization may get slightly worse pull requests without pages, failing tests, or another obvious signal. We need a way to connect an `AGENTS.md` change to its impact.

## Measurement

`AGENTS.md` changes should be A/B tested. Measure:

- Test pass rate
- Agent diff equivalence to a human diff
- How much extra code the agent writes relative to the human diff

The goal is to measure how well the agent is aligned with your codebase and how closely its output matches your expectations.

Our practical approach is an [instruction eval](https://www.stet.sh/instruction-evals): replay real repository tasks, compare the current `AGENTS.md` with a candidate, and inspect whether quality improved. The same [how Stet measures real AI coding performance](https://www.stet.sh/methodology) loop also applies to model, harness, and reasoning-level changes.

## Monitoring

Agents change over time. A silent update from Anthropic or OpenAI can shift behavior even when your `AGENTS.md` has not changed. Without monitoring, you cannot know what happened—the environment shifted underneath it.

Organizations need continuous monitoring, not only pre-release testing. We compare this with monitoring production after deployment rather than trusting green CI forever.

## Reviews

Review `AGENTS.md` with the same rigor as critical production code. Read each change carefully and have a human approve it. Poor instructions can scale across every engineer in an organization, making the file’s quality consequential.

## Conclusion

When we measured agent quality beyond test pass rate, we found a **2x quality gap** hiding in plain sight: [Your AI coding benchmark is hiding a 2x quality gap](https://www.stet.sh/blog/both-pass).

If you are not measuring and releasing `AGENTS.md` changes deliberately, you are guessing at scale. Without a deliberate release process, you have a hope-based one.

## References

- [HumanLayer: Skill issue: harness engineering for coding agents](https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents)
- [HumanLayer: Writing a good CLAUDE.md](https://www.humanlayer.dev/blog/writing-a-good-claude-md)

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fagents-md-testing&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fagents-md-testing). Opening a form does not execute a tool.