llms.txt
+22 −12
https://agenticwebscore.com/llms.txt
− AgenticWebScore measures public websites from the outside and scores how well they serve agents. Rubric v0.9 defines 60 checks across 5 dimensions: 36 are scored and 24 are unscored. The score is 70% web standards (28 scored checks) and 30% agent conventions (8). When no convention applies or none was measured, the score is the web-standards part alone. An unscored check earns no points and never changes the score or the level. It refuses to score what it cannot measure: a site that blocks every− - [Learn](https://agenticwebscore.com/learn): 23 source-cited articles on agent readiness.− - [ard.json: the Agentic Resource Discovery manifest and the ai-catalog.json rename](https://agenticwebscore.com/learn/ard-json-agentic-resource-discovery): ARD v0.91 asks sites to serve /.well-known/ard.json instead of ai-catalog.json. The fields, what Lighthouse 13.5 reads, and how this scanner checks it.− - [Personal AI agents with no user agent: what robots.txt cannot block and what a site can do instead](https://agenticwebscore.com/learn/personal-ai-agents-no-user-agent): Meta documents no user agent for Muse, which drives a real browser, so robots.txt cannot name it. What Amazon and Wikimedia saw and what a site can still do.− - [Agentic commerce protocols compared: AP2, ACP, UCP, x402 and MPP](https://agenticwebscore.com/learn/agentic-commerce-protocols): What AP2, ACP, UCP, x402 and MPP each do, which layer of a transaction each occupies, who governs it, what the scanner detects, and when to implement none.− - [Bing and Copilot: what noindex, noarchive, nocache, nosnippet and data-nosnippet do to AI answers](https://agenticwebscore.com/learn/bing-copilot-noarchive-nocache): What each Bing directive does to search results, Copilot answers and grounding, as Bing's own guidelines state it, and how Google reads the same tags.− - [Content-Signal in robots.txt: search, ai-input and ai-train explained](https://agenticwebscore.com/learn/content-signal-robots-txt): What the search, ai-input and ai-train values of a Content-Signal line mean, where the line goes, who reads it today, and how it differs from IETF aipref.− - [HTTP caching for crawlers: ETag, Last-Modified and 304](https://agenticwebscore.com/learn/http-caching-crawlers): Why Googlebot and Bingbot send If-None-Match, how to test a 304 Not Modified with curl, and the CDN behaviours that quietly break validators on a page.− - [One rubric, 8 site profiles: how the site profile changes which checks apply](https://agenticwebscore.com/learn/site-profiles-methodology): A site profile decides which checks an agent-readiness scan asks of a shop, a publisher or an API. How 8 site profiles are detected, weighted and shown.− - [What 5 well-known sites serve to AI agents (October 2026)](https://agenticwebscore.com/learn/five-sites-agent-readiness-october-2026): Scores, levels and the concrete reasons behind them for Vercel, Cloudflare, Resend, GitHub and MDN, from the scan records of 20 October 2026.− - [llms-full — /llms-full.txt expanded context (agent convention)](https://agenticwebscore.com/checks/llms-full): A convention that puts a site's text in one file. The llms.txt proposal does not define it.− - [mpp-discovery — MPP discovery in OpenAPI (x-payment-info) (unscored)](https://agenticwebscore.com/checks/mpp-discovery): An operation with x-payment-info tells an agent that the call is paid, and states the intent, the payment method, and the amount. MPP is a draft, and charging agents is a business decision.+ AgenticWebScore measures public websites from the outside and scores how well they serve agents. Rubric v0.9.1 defines 60 checks across 5 dimensions: 36 are scored and 24 are unscored. The score is 70% web standards (28 scored checks) and 30% agent conventions (8). When no convention applies or none was measured, the score is the web-standards part alone. An unscored check earns no points and never changes the score or the level. It refuses to score what it cannot measure: a site that blocks eve+ - [Learn](https://agenticwebscore.com/learn): 33 source-cited articles on agent readiness.+ - [Agentic commerce protocols compared: AP2, ACP, UCP, x402 and MPP](https://agenticwebscore.com/learn/agentic-commerce-protocols): What AP2, ACP, UCP, x402 and MPP each do, which layer of a transaction each occupies, who governs it, what the scanner detects, and when to implement none.+ - [ard.json: the Agentic Resource Discovery manifest and the ai-catalog.json rename](https://agenticwebscore.com/learn/ard-json-agentic-resource-discovery): ARD v0.91 asks sites to serve /.well-known/ard.json instead of ai-catalog.json. The fields, what Lighthouse 13.6 reads, and how this scanner checks it.+ - [Bing and Copilot: what noindex, noarchive, nocache, nosnippet and data-nosnippet do to AI answers](https://agenticwebscore.com/learn/bing-copilot-noarchive-nocache): What each Bing directive does to search results, Copilot answers and grounding, as Bing's own guidelines state it, and how Google reads the same tags.+ - [Content-Signal in robots.txt: search, ai-input and ai-train explained](https://agenticwebscore.com/learn/content-signal-robots-txt): What the search, ai-input and ai-train values of a Content-Signal line mean, where the line goes, who reads it today, and how it differs from IETF aipref.+ - [HTTP 402 and x402: how a site charges agents per request, and what a scanner can see](https://agenticwebscore.com/learn/http-402-x402-pay-per-crawl): What HTTP 402 means, what x402, MPP and Cloudflare's Pay Per Crawl send, how pay per use differs, and which of them the scanner can detect.+ - [HTTP caching for crawlers: ETag, Last-Modified and 304](https://agenticwebscore.com/learn/http-caching-crawlers): Why Googlebot and Bingbot send If-None-Match, how to test a 304 Not Modified with curl, and the CDN behaviours that quietly break validators on a page.+ - [llms.txt examples: 6 real files from Stripe, Anthropic, Vercel, Resend, GitHub and Cloudflare, annotated](https://agenticwebscore.com/learn/llms-txt-examples): 6 real /llms.txt files, read on 31 October 2026 and checked against the llmstxt.org format: Stripe, Anthropic, Vercel, Resend, GitHub and Cloudflare.+ - [llms.txt vs llms-full.txt vs AGENTS.md vs SKILL.md vs MCP server card: which file does which job](https://agenticwebscore.com/learn/llms-txt-vs-agents-md-vs-skill-md): llms.txt lists pages for agents. llms-full.txt, AGENTS.md, SKILL.md, a server card and robots.txt do other jobs. Who reads each, where it lives, what we check.+ - [Migrating an MCP server to 2026-07-28: no initialize, no sessions, server/discover](https://agenticwebscore.com/learn/mcp-2026-07-28-migration): MCP 2026-07-28 drops initialize and Mcp-Session-Id and requires server/discover. A checklist, curl tests for both revisions, and how this scanner scores it.+ - [NLWeb explained: what it is, how it relates to MCP, and what a site exposes](https://agenticwebscore.com/learn/nlweb-explained): NLWeb adds an ask endpoint to a site, answers in Schema.org JSON, and doubles as an MCP server. What it is, how it differs from MCP, and what a scan sees.+ - [One rubric, 8 site profiles: how the site profile changes which checks apply](https://agenticwebscore.com/learn/site-profiles-methodology): A site profile decides which checks an agent-readiness scan asks of a shop, a publisher or an API. How 8 site profiles are detected, weighted and shown.+ - [Retry-After and 429 for AI crawlers: slow them down instead of blocking them](https://agenticwebscore.com/learn/retry-after-429-ai-crawlers): How to slow an AI crawler with 429 or 503 and Retry-After instead of blocking it, what Google, Bing and Cloudflare document, and nginx and middleware code.+ - [UCP manifest: what to publish at /.well-known/ucp, with an annotated example](https://agenticwebscore.com/learn/ucp-manifest-well-known-ucp): What a Universal Commerce Protocol profile at /.well-known/ucp contains, an annotated example, the hosting rules, and what the scanner reads.+ - [What 5 well-known sites serve to AI agents (October 2026)](https://agenticwebscore.com/learn/five-sites-agent-readiness-october-2026): Scores, levels and the concrete reasons behind them for Vercel, Cloudflare, Resend, GitHub and MDN, from the scan records of 20 October 2026.+ - [What does Google-Extended block? Gemini training and grounding, not Search or AI Overviews](https://agenticwebscore.com/learn/google-extended-what-it-blocks): Google-Extended limits Gemini training and grounding. It does not control Google Search or AI Overviews, which have their own Search Console control.+ - [What is a good agent readiness score? 5 well-known sites scored 61.7–88.5](https://agenticwebscore.com/learn/agent-readiness-score-what-is-good): No agent readiness score is good on its own. 5 well-known sites scored 61.7–88.5 in November 2026, and the level says more than the number does.+ - [How to find AI crawlers in your server logs, and tell real ones from fakes](https://agenticwebscore.com/learn/find-ai-crawlers-in-server-logs): grep and awk recipes to find GPTBot, ClaudeBot and other AI crawlers in nginx and Apache logs, then check each claim against IP lists, DNS or signatures.+ - [Personal AI agents with no user agent: what robots.txt cannot block and what a site can do instead](https://agenticwebscore.com/learn/personal-ai-agents-no-user-agent): Meta documents no user agent for Muse, which drives a real browser, so robots.txt cannot name it. What Amazon and Wikimedia saw and what a site can still do.+ - [llms-full — llms-full.txt expanded context (agent convention)](https://agenticwebscore.com/checks/llms-full): A convention that puts a site's text in one file. The llms.txt proposal does not define it.+ - [mpp-discovery — MPP discovery (x-payment-info or a 402 Payment challenge) (unscored)](https://agenticwebscore.com/checks/mpp-discovery): An operation with x-payment-info tells an agent that the call is paid, and states the intent, the payment method, and the amount. A 402 answer with a WWW-Authenticate Payment challenge is MPP's own way to ask for payment. MPP is a draft, and charging agents is a business decision.