Yesterday, Bookmark Conversations, the little application I’ve been building to find unexpected connections between my bookmarked articles (see more https://justme.website/content-creation/read-later-more-like-read-never-so-i-built-bookmark-conversations/), came up with something very interesting. It connected a Cloudflare article about the growing amount of automated traffic on the internet with a technical specification from SPUR, an initiative exploring how publishers can track the use of their content by AI systems.
At first, the connection seemed fairly straightforward. If AI agents are becoming increasingly active on the web, publishers need to know who these agents are and what they’re doing with their content. But the more I looked into it, the more fascinating it became. Because there are actually two very different questions here:
How do we know who an AI agent really is?
And how can we verify what it does with the information it collects?
It turns out that quite a few people are working on the first question. The second is considerably more difficult.
Could I see your passport, please?
Imagine someone walking into a library, picking up a book and introducing themselves as a researcher from a well-known university. The librarian might be perfectly happy to help. But how does he or she know the visitor really works for that university?
Something similar happens constantly on the web. Software visiting a website can identify itself through a user-agent string. A crawler might claim to be Google’s crawler, for example, or another well-known automated service. But simply claiming an identity is not the same as proving it. And as AI agents begin doing more than collecting information, this distinction becomes increasingly important. Some agents browse websites, compare products, book services or carry out tasks on behalf of users. Others collect material that may eventually appear in AI-generated answers.
So how do we distinguish a legitimate agent from software pretending to be one? One answer is Web Bot Auth, an emerging approach being developed through the Internet Engineering Task Force (IETF) (https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/). Instead of simply announcing its identity, an automated visitor can cryptographically sign its requests. A website can then verify that the request was signed using a particular private key. It’s a little like replacing a handwritten name badge with a digitally verifiable identity document.
Of course, proving that an agent controls a particular cryptographic key doesn’t automatically tell us whether that agent is trustworthy. But it gives us something much more useful than a name that anybody can type into an HTTP request. Cloudflare is among the companies exploring and implementing this approach.
The surprising part: payment companies are interested too
I initially thought this was mainly a problem for website owners and publishers. But then I discovered that payment companies are working on remarkably similar questions. Visa has developed its Trusted Agent Protocol (https://developer.visa.com/docs/trusted-agent-protocol/trusted-agent-protocol-specifications), intended to help merchants recognise AI agents involved in shopping transactions. Mastercard is also exploring ways to establish what a customer has actually authorised an AI agent to do.
Here is what they are trying to do. If an AI assistant buys something on your behalf, a shop needs more than an assurance that the visitor is an AI agent. It needs to know who operates the agent, whether the agent is acting for a genuine customer and whether that customer authorised the purchase. That’s quite a different problem from identifying a web crawler. Yet the underlying questions are more or less the same: Who are you? Who sent you? What are you allowed to do? And can you prove it?
The FIDO Alliance (https://fidoalliance.org/), known for its work on authentication and passkeys, is also involved in discussions about securing interactions involving AI agents.
I find this development rather fascinating. We seem to be watching the early construction of something resembling a digital identity infrastructure for software that acts on its own. Not a single universal passport system, at least not yet. More a collection of standards, protocols and experiments that may eventually fit together.
And that raises another question I hadn’t initially considered: who will issue these identities, and who gets to decide which agents are trusted? I’d much prefer an open system in which a small independent website can verify an agent without depending entirely on a large technology company or government agency. Otherwise, we risk creating another layer of gatekeepers on the internet.
But what happens after the agent leaves?
This brings me back to the second part of the Bookmark Conversations discovery. Let’s imagine that an AI agent visits a news website. It identifies itself cryptographically. The publisher verifies its identity and allows it to retrieve an article.
So far, so good. But what happens next? Does the agent simply read the article to answer a question? Does it store a copy? Does it pass the information to another system? Does the article influence a generated answer without ever being quoted or linked? The publisher can’t necessarily see any of this. And that becomes particularly interesting when publishers want to license their content to AI companies.
Several initiatives are trying to create the infrastructure for such arrangements. Really Simple Licensing (https://rslstandard.org/) focuses on machine-readable licensing conditions. Cloudflare has been experimenting with Pay Per Crawl, allowing website owners to charge automated visitors for access (https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/).
SPUR (https://github.com/SPUR-Coalition/telemetry) takes another approach. Its Content Telemetry specification describes ways to report how content moves through AI systems and how it is used. When I examined SPUR more closely, I discovered that its specification already considers verified bot identities. So the original Bookmark Conversations discovery wasn’t entirely correct in suggesting that identity verification had simply been overlooked.
But it had uncovered a much more interesting issue: Knowing who collected an article is not the same as knowing what happened to that piece of content afterwards. An AI company might report that an article was used in a particular way. But can the publisher independently verify that report? That’s where things become difficult.
A receipt isn’t necessarily proof
Suppose a publisher receives a report saying that an article was retrieved 500 times and used in 30 AI-generated answers. That’s useful information. But where did those numbers come from? If they were reported by the AI provider itself, the publisher still has to trust that provider’s reporting system. Even a cryptographically signed report only proves who signed it and that it hasn’t subsequently been altered. It doesn’t automatically prove that every relevant use was recorded.
This reminds me a little of advertising measurement. For years, advertisers and publishers have wrestled with questions about impressions, clicks, attribution and independent verification. AI introduces an additional complication. An article might influence an answer without appearing in the final response. Information might pass through several services. And the distinction between retrieving content, processing it and actually using it becomes surprisingly difficult to define.
Perhaps some of these events can eventually be recorded in independently auditable systems. Perhaps trusted execution environments, cryptographic proofs or third-party measurement services will play a role. But none of these approaches should be mistaken for a complete solution. Proving what happened inside a complex AI system is very different from proving that a particular computer sent a particular HTTP request. And I suspect this distinction will become increasingly important as the market for licensing content to AI companies develops.
Two pieces of a much bigger puzzle
Interestingly, these developments are coming from different corners of the internet. Security researchers are thinking about cryptographic identities. Payment companies are working on agent authorisation. Publishers are experimenting with licensing and usage reporting. Internet standards organisations are trying to make sure the underlying technologies can work together. They aren’t necessarily building one coordinated system. But they are encountering different parts of the same problem.
The internet is gradually filling with software that can act on behalf of people and organisations. We need ways to recognise that software, establish what it is allowed to do and hold it accountable for its actions. A digital passport may help with the first part. It doesn’t automatically solve the last. And there is a rather uncomfortable possibility here: we might eventually become very good at identifying the AI agents that visit our websites while remaining surprisingly dependent on their operators to tell us what happens to our content.
One discovery, quite a few new questions
I like that this whole line of thought began with two bookmarks that decided were related. The application didn’t provide a definitive answer. In fact, one part of its original interpretation needed correcting after I checked the sources. But that’s fine. I didn’t build it to deliver unquestionable conclusions. I built it to notice connections I might otherwise miss and give me something worth investigating.
And this one certainly did. I started with a question about a technical specification for AI content telemetry. I ended up wondering whether we’re witnessing the birth of an identity system for autonomous software, and whether the people creating the content these systems consume will ever be able to verify how that content is used. And get paid for it. Not bad for a couple of bookmarks that might otherwise have remained in my read later collection forever.