llms.txt
+2 −2
https://doorman.clawhard.com/llms.txt
− - `GET https://doorman.clawhard.com/api/v1/check`: Reads the public robots.txt of a website and reports which AI bots it lets in, by purpose: AI training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more), AI search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot), assistants opening pages for a person (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User) and classic search engines (Googlebot, Bingb− Yes. There is a public JSON API (GET https://doorman.clawhard.com/api/v1/check?url=example.com, see /openapi.json) and the tools doorman_check_ai_bot_access, doorman_check_robots_txt and doorman_write_robots_txt on the Clawhard MCP server at https://mcp.clawhard.com/mcp. Doorman has its own MCP server at https://doorman.clawhard.com/mcp (Streamable HTTP, no key, no login). Add it to Claude, ChatGPT, Cursor, VS Code and other MCP clients, or use the whole studio at https://mcp.clawhard.com/mcp.+ - `GET https://doorman.clawhard.com/api/v1/check`: Reads the public robots.txt of a website (or robots.txt text you pass as robots_txt) and reports which AI bots it lets in, by purpose: AI training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more), AI search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot), assistants opening pages for a person (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User) + Yes. There is a public JSON API (GET https://doorman.clawhard.com/api/v1/check?url=example.com, see /openapi.json) and the MCP tool doorman_check_ai_bot_access (pass a url, or robots_txt text) on https://doorman.clawhard.com/mcp and the Clawhard MCP server at https://mcp.clawhard.com/mcp. Writing a robots.txt is on the website and in the API (/api/v1/build).