AI Crawler Checker — Robots.txt Policy for Six Bots for Windsurf

io.github.kriton-dev/ai-crawler-access-check

Check declared robots.txt permissions for six AI crawlers at URL paths

client:Windsurf transport:streamable-http

Install AI Crawler Checker — Robots.txt Policy for Six Bots in Windsurf

~/.codeium/windsurf/mcp_config.json · windows: %USERPROFILE%\.codeium\windsurf\mcp_config.json

{
  "mcpServers": {
    "ai-crawler-access-check": {
      "serverUrl": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"
    }
  }
}

Global only — Windsurf has no per-project MCP config. Refresh Cascade after saving. Windsurf docs