AI Agent 时代需要新网络规则：论文提出 agents.txt 协议

This paper says the web needs new rules because AI agents now read websites for people.

The problem is that today’s web still assumes a human is looking at each page, seeing ads, clicking links, and reading visual layouts.

AI agents break that setup because they can collect and summarize content without sending people back to the original sites, which hurts publishers and makes websites block them.

The authors propose treating a helpful AI agent like a human’s proxy, so it should get similar access as that person, but with clear identity, purpose, limits, and payment rules.

They propose adding a new “agent metadata” layer to normal web requests, where an AI agent tells a website who it is, which human it represents, and why it wants the content.

The website then uses a new policy file called agents.txt to decide what to do: allow it, rate-limit it, charge tokens, inherit the user’s subscription, serve agent-friendly content, or block bad behavior.

They also want content to carry provenance tags, so agents can tell whether something was made by a human, AI, or both.

Without a new setup, the web may become harder for agents to access, worse for publishers to fund, and less reliable as AI-made content feeds more AI-made content.

Link – arxiv. org/abs/2606.19116

Title: "Towards an Agent-First Web: Redesigning the Web for AI Agents"