Cloudflare launches an open registry to authenticate web bots
Cloudflare has unveiled an ambitious proposal that could transform how web platforms identify and manage automated traffic.
The American company — a cornerstone of the global internet infrastructure — aims to introduce an open registry format to authenticate bots and web agents, employing a decentralized approach based on verifiable cryptographic keys, rather than relying on traditional IP address identification systems.
Why is this change necessary?
Historically, servers identified bots based on their IP addresses, a mechanism effective in the era of fixed networks. However, today, intelligent agents and AI bots — including search engines, assistants, and commercial crawlers — utilize shared infrastructures, proxies, or dynamic IP addresses.
The consequence is:
- False positives, where legitimate bots are blocked,
- and false negatives, where malicious bots slip through the cracks.
Cloudflare believes it is time to adopt a more reliable, open, and verifiable system.
How does the new format work?
The open registry proposed by Cloudflare would allow bot creators to publish verifiable public keys in a standardized manner. These keys would be accessible via plain text files hosted on public repositories (such as GitHub) or distributed via email, following a format inspired by web standards.
For example, a ChatGPT bot could list its key at https://chatgpt.com/.well-known/agent-registry-keys. A web server could then check this URL to confirm that the traffic originates from the declared bot, without relying on a central authority.
In essence: instead of relying on an agent’s IP address, we verify its cryptographic identity.
A decentralized and interoperable model
This approach marks a departure from proprietary lists and closed filters previously used. Anyone — from independent developers to large corporations — could maintain their own registry, which would be interoperable with others.
The benefits are numerous:
- Online commerce: clearly distinguish legitimate purchase bots from malicious scrapers.
- Research and AI: prevent mistakenly blocking academic crawlers.
- Interoperability: enable platforms to automatically validate external agents.
Cloudflare has already announced it is working with Visa and Mastercard to apply this principle to securing autonomous commercial agents.
A continuation of the work on Web Bot Auth
This project builds on Cloudflare’s previous efforts surrounding Web Bot Auth, a cryptographic authentication system for bots launched earlier. The difference here is that the registry aims to standardize and open the process for the entire internet ecosystem, beyond just Cloudflare’s services.
Cloudflare presents this innovation as a step toward a universal trust infrastructure, founded not on the reputation of an IP but on a verified digital identity. This could reduce abusive blockages, streamline exchanges between sites and bots, and enhance overall web security.
“The future of automated traffic must rely on cryptography, not on assumption,” sums up a Cloudflare engineer cited in the report.
Upcoming challenges
Experts applaud this initiative but note several challenges:
- Community adoption: the format must become a recognized standard, not merely a Cloudflare tool.
- Governance: who will arbitrate in cases of conflicts between keys or abuse?
- Interoperability: convincing major web players (Google, Amazon, Meta, etc.) to adopt the same protocol will be essential.
Staying true to its open-source DNA, Cloudflare will publish the format on GitHub (github.com/cloudflare/agents), hoping to build a community of developers and researchers around it.
By launching this registry of verifiable agents, Cloudflare is not just implementing a technical update: it proposes a new trust paradigm on the internet, where every bot, every agent, and every AI assistant can transparently and universally prove its identity.
A small cryptographic revolution for a safer, more open, and more collaborative web.
