AI Crawler Checker — Robots.txt Policy for Six Bots for Claude Desktop
io.github.kriton-dev/ai-crawler-access-check
Check declared robots.txt permissions for six AI crawlers at URL paths
client:Claude Desktop
transport:streamable-http
Install AI Crawler Checker — Robots.txt Policy for Six Bots in Claude Desktop
~/Library/Application Support/Claude/claude_desktop_config.json · windows: %APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"ai-crawler-access-check": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"
]
}
}
}Claude Desktop has no remote transport in its config file, so this bridges through mcp-remote. Restart the app after saving. Claude Desktop docs