What is Web Bot Auth?
Web Bot Auth is a way for a bot to sign its HTTP requests so that the site receiving them can verify who sent them. That is the whole idea, and it is worth stating plainly because the problem it replaces is embarrassingly weak. Until now, the only thing an automated client offered as identification was a user-agent string, which is a header any client can set to any value. Verifying it meant reverse DNS lookups against published IP ranges, which is slow, brittle and unavailable to most site operators. Web Bot Auth swaps that for asymmetric cryptography. The operator holds a private key, the site fetches the matching public key from a location the operator controls, and a signature either validates or it does not. There is no judgment call and no IP list to maintain. The practical consequence is that "are you really OpenAI" stops being a guess. It also means a site can finally make a policy decision it could not make before: serve verified agents, and rate limit everything that will not identify itself.
How Web Bot Auth works, end to end
Four steps. The operator generates a keypair, in practice Ed25519, and publishes the public half as a JSON Web Key Set at /.well-known/http-message-signatures-directory on a hostname it controls. For every outbound request, the agent computes a signature over a defined set of message components and attaches two headers defined by RFC 9421: Signature-Input, which names the components covered and the signature parameters, and Signature, which carries the bytes. It is also recommended to send a Signature-Agent header pointing at the directory, and when that header is present it must itself be one of the signed components, so an attacker cannot redirect a verifier to a key directory they control. The receiving site resolves the key by its keyid, which is the base64url SHA-256 thumbprint of the JWK, recomputes the signature base, and checks it. Cloudflare, AWS WAF and other edge providers do this for you and hand your application a label. Doing it yourself is a handful of lines with an off-the-shelf library, and Cloudflare publishes both a Rust crate and a TypeScript npm package named web-bot-auth.
Which AI agents actually publish Web Bot Auth keys?
This is where the coverage stops and the useful part starts. Every explainer asserts that the major AI companies are signing their traffic. We checked. On September 1, 2026 we requested /.well-known/http-message-signatures-directory from more than twenty hostnames belonging to the largest AI operators, search engines, browser-agent vendors and infrastructure providers, following redirects and inspecting the response content type. Four returned a valid signed-key directory: chatgpt.com, meta.com, you.com and browserbase.com. The full table is below, and three findings in it are worth acting on. First, the directory lives on the hostname the traffic comes from, not the corporate domain: openai.com returns 404 while chatgpt.com serves a valid directory. Second, an HTTP 200 does not mean a directory exists, because single-page apps happily return an HTML shell for any path. platform.openai.com returned 200 with text/html, and our first pass wrongly recorded shopify.com the same way before a careful re-probe on September 2, 2026 found a genuine directory behind a redirect to www. Check the content type, and follow redirects. Third, and most useful: a 404 does not prove an operator is unsigned, because Cloudflare and AWS both let an operator register a key directory at an arbitrary URL rather than the well-known path.
A detail nobody mentions: these keys expire, and soon
OpenAI is the only operator in our sample that publishes full key metadata, and reading it is instructive. The ChatGPT directory serves a single Ed25519 key with a kid, use: sig, a not-before timestamp of January 1, 2025 and an expiry we have now watched move. When we first probed on September 1, 2026 the key was set to expire on September 8, 2026 at 06:58 UTC, roughly a week out. On September 3, 2026 we re-fetched the same directory and the expiry had been pushed to September 10, 2026 at 09:17 UTC under the same kid. We fetched it a third time on September 16, 2026 and the same key had been extended again, to September 23, 2026 at 04:58 UTC, still under the identical kid of otMqcjr17mGyruktGvJU8oojQTSMHlVm7uO-lrcqbdg and still with nbf pinned to January 1, 2025. Three readings, three extensions, one unchanged key identifier: OpenAI is rolling the expiry forward roughly weekly rather than publishing a new key. The practical lesson for anyone building a verifier is that kid is stable and exp is not, so cache on the key identifier and re-fetch the directory on a timer rather than trusting an expiry you read once. That is a short-dated credential being extended in place, and it is the clearest evidence we have that this is live rotating infrastructure rather than a value you can read once and hard-code. This is normal and correct behaviour, and it is also the single most likely way a Web Bot Auth deployment breaks in production. If you cache a verifier key indefinitely, your allowlist silently starts rejecting the exact agent traffic you spent a quarter trying to attract, and the failure looks like a traffic drop rather than an error. AWS WAF models this explicitly with an expired label distinct from invalid, which tells you the design assumption: keys rotate, verifiers re-poll. Treat the directory endpoint as a production dependency on both sides. If you are the operator, your directory going down means every verifier fails you closed, so it belongs in the same monitoring you already run on your API.
What a compliant Web Bot Auth signature must contain
The architecture draft is specific about the signature parameters, and this is the part implementers get wrong. An agent must cover either @authority or @target-uri, and must include created, expires, keyid, and a tag whose value is exactly web-bot-auth. A nonce of 64 random base64url-encoded bytes is recommended and is what makes replay expensive. The tag matters more than it looks: it is the field that separates general-purpose message signing from bot identification, and Visa Trusted Agent Protocol reuses exactly this mechanism with its own tag values to distinguish an agent that is only browsing from one carrying a payment instruction. Two hard prohibitions are worth repeating because they close real attacks. Implementations must not use shared HMAC, since a symmetric secret shared with every verifier is not an identity. And keys must not be tied to a specific human individual, because the signature identifies a bot operator, not a person, and conflating those turns a routing header into personal data.
Is Web Bot Auth an IETF standard yet?
No, and the honest answer here matters because vendors are already selling it as one. There is a real IETF working group: Web Bot Authentication (webbotauth), in the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef, operating under an approved charter. Its milestones called for standards-track authentication specifications to reach the IESG by April 30, 2026 and a Best Current Practice operational document by August 31, 2026. What has not happened is adoption of a document. The specifications people are implementing are individual submissions authored by Thibault Meunier of Cloudflare and Sandor Major of Google. The original architecture draft, draft-meunier-web-bot-auth-architecture-05 of March 2, 2026, has expired and been replaced by draft-meunier-webbotauth-httpsig-protocol, at version 02 as of August 18, 2026, which still carries the standard disclaimer that it has no formal standing in the IETF standards process. So the accurate statement is that Web Bot Auth shipped into production at several of the largest networks on the internet before it became a standard, and the wire format could still change.
Web Bot Auth on Cloudflare
Cloudflare is the origin of the protocol and the reason it has any adoption at all. On July 1, 2025 it added HTTP Message Signatures to its Verified Bots Program, the same day it began blocking AI crawlers by default for new zones. Those two decisions belong together: Cloudflare closed the door and simultaneously handed out a key. Registration is deliberately simple. An operator opens the Verified Bots submission form, selects Request Signature as the verification method, gives the URL of its key directory, hosts a JWKS there and signs requests for that URL using one of the published libraries. Cloudflare then validates signatures at its edge and exposes the result to firewall rules through cf.verified_bot_category, so a site owner writes a rule rather than any code. Note the flexibility that our probe results depend on: because the operator supplies the directory URL, the well-known path is a convention rather than a requirement, which is why a 404 at the conventional location is weak evidence of anything.
Web Bot Auth on AWS WAF
AWS announced Web Bot Auth support in AWS WAF on November 21, 2025, and its implementation is the clearest published model of how verification should behave in the failure cases. AWS polls registered operator key directories, maintains a key registry, checks each signature and then labels the request rather than making the decision for you. The labels are the interesting part: verified when the signature validates against a known public key, invalid when a signature is present but cryptographic validation fails, expired when the key used has passed its validity window, and unknown_bot when the key ID is not in the registry at all. Those four states are worth copying into your own logic even if you never touch AWS, because collapsing them into a boolean is how teams end up blocking a legitimate partner over a key rotation. Verified Web Bot Auth traffic is allowed by default in Bot Control. Amazon has also wired the protocol into the agent side: Bedrock AgentCore uses Web Bot Auth in its managed browser to reduce CAPTCHA challenges, which is the same idea pointed the other way, and it pairs with what we cover in Bedrock AgentCore Payments.
Web Bot Auth on Shopify, and why merchants suddenly care
Shopify made this commercial, and it practices what it enforces: a re-probe on September 2, 2026 found shopify.com serving a valid Ed25519 key directory, behind a redirect to www, with the correct content type and a validity window running from March 13, 2026 to March 13, 2027. Our September 1 pass had recorded that host as an HTML false positive, which is a useful reminder that redirect handling changes the answer. The commercial part is the policy. On May 30, 2026 its developer changelog told bots and agents to identify themselves via Web Bot Auth, and stated that agents which do not sign their requests are subject to the strictest rate limits on the Storefront API and on Shopify-hosted online store pages. Read that as a buyer rather than as an engineer. If you operate a shopping or procurement agent, an unsigned agent now shares a throttle bucket with anonymous scrapers across a very large share of US online retail, and your product gets slower on exactly the merchants your users care about. If you are the merchant, you have gained a lever you did not have last year: you can serve agent traffic generously without also serving everyone who claims to be an agent. This is the pattern to expect everywhere. Signature becomes the price of admission to good service, and the platforms that hold checkout are the ones setting it. The seller side of that same platform is covered in our Shopify agentic commerce guide.
How Web Bot Auth relates to agentic checkout and Visa TAP
Web Bot Auth is the identity floor that the payment protocols are being built on top of. The Visa Trusted Agent Protocol, announced with Cloudflare on October 14, 2025, is built on RFC 9421 HTTP Message Signatures and explicitly aligned with Web Bot Auth, extending it with tag values that distinguish an agent that is merely browsing from one that carries a consumer-authorized payment instruction, and with the ability to convey Payment Account References. Our walkthrough for sellers is in the Visa Trusted Agent Protocol guide for merchants. The layering is clean once you see it. Web Bot Auth answers which bot is this. TAP adds and does it have a payment mandate. AP2 answers did a human authorize this specific purchase. ACP answers how does the order get placed. A2A answers how do two agents talk at all. Every one of those is a merchant-side or network-side question, which is exactly why the buy side keeps finding a gap.
What Web Bot Auth does not decide
A verified signature tells a merchant that a request genuinely came from a known agent operator. It does not tell anyone whether the purchase behind that request was a good idea. There is no budget in the specification, no cumulative spending cap, no counterparty allowlist, no velocity rule and no approval threshold, and there was never going to be, because those are properties of the company deploying the agent rather than of the HTTP request. The uncomfortable version for a buyer is this: Web Bot Auth makes your agent more trusted, not more controlled. Signing your traffic gets you past the rate limiter and into the fast lane at Shopify, Cloudflare and AWS, which means a misbehaving agent now executes its mistake faster and with fewer obstacles than an unsigned one would have. That is a net win only if the control lives somewhere. The same gap shows up in every agentic payment standard, and the layer that closes it is agent governance policy sitting above the rail.
Implementing Web Bot Auth as a bot operator
Six things, in the order they bite. Generate an Ed25519 keypair and keep the private key in whatever your platform uses for secrets, never in the repository. Publish the public key as a JWKS at /.well-known/http-message-signatures-directory on the hostname your requests actually originate from, since that is what verifiers will look at, and include kid, use, nbf and exp so verifiers can reason about rotation. Sign every outbound request with the required components and the web-bot-auth tag, using the Rust or TypeScript library rather than hand-rolling the signature base, which is where the subtle bugs live. Send Signature-Agent and make sure it is covered by the signature. Register the directory with Cloudflare Verified Bots and with AWS, because edge registration is what converts a valid signature into actual preferential treatment. Then plan rotation before you need it: overlap the old and new keys in the directory, monitor that the endpoint stays reachable, and treat expiry dates as calendar events, not implementation details.
Verifying Web Bot Auth as a merchant or API owner
If you sit behind Cloudflare or AWS WAF, most of this is a configuration change and you should start there rather than writing code. Turn on the verified-bot signal, then write the policy you actually want, which is usually three tiers: verified agents from operators you have chosen get normal or elevated limits, verified agents you do not recognise get a reduced but workable allowance, and unsigned automated traffic gets the strict bucket. Log the verification state on every request, including which operator, because that log is the only record you will have when someone asks who bought this. If you are verifying yourself, cache directories with a short TTL, honour expires, reject a missing or unsigned Signature-Agent, and enforce the nonce window so a captured signature cannot be replayed. One thing to decide deliberately rather than by accident: whether verification is a gate or a signal. Treating it as a hard gate today will block real customers, since as our probe shows most operators are not signing yet.
How Agentspay governs what a verified agent is allowed to buy
Agentspay is the buy-side control plane that sits above whatever identity and payment standard you adopt. Web Bot Auth proves the agent is who it claims to be; Agentspay decides whether this agent, right now, should be permitted to spend this amount with this counterparty. Every agent gets its own funded wallet instead of shared access to a company credential, and every intended purchase is checked against policy before a payment credential is ever issued, covering the per-transaction ceiling, the cumulative budget over a window, the merchant allowlist and the velocity rule that catches a retry loop. Anything above your threshold pauses for human approval rather than settling. What the agent receives is a scoped credential such as a merchant-locked virtual card, and every decision lands in an immutable audit trail naming the agent, its human owner, the intent and the policy that allowed it. That identity chain is the same one Know Your Agent describes, and because the policy layer is rail-neutral, signing your traffic with Web Bot Auth changes nothing about how your controls are written.