AI Crawler Checker — Robots.txt Policy for Six Bots for Claude.ai

io.github.kriton-dev/ai-crawler-access-check

Check declared robots.txt permissions for six AI crawlers at URL paths

client:Claude.ai transport:streamable-http

Install AI Crawler Checker — Robots.txt Policy for Six Bots in Claude.ai

Settings → Connectors → Add custom connector

https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check

Claude on the web takes a remote URL and completes OAuth in the browser. For a packaged server use Claude Desktop or Claude Code instead. Claude.ai docs

AI Crawler Checker — Robots.txt Policy for Six Bots in other clients

Claude Code
Claude Desktop
Cursor
VS Code
Zed
Windsurf
Cline
Gemini CLI
Grok
ChatGPT
LangChain

AI Crawler Checker — Robots.txt Policy for Six Bots MCP server