Skip to content
ChatGPT

AI Crawler Checker — Robots.txt Policy for Six Bots for ChatGPT

io.github.kriton-dev/ai-crawler-access-check

Check declared robots.txt permissions for six AI crawlers at URL paths

client:ChatGPT transport:streamable-http

Install AI Crawler Checker — Robots.txt Policy for Six Bots in ChatGPT

Settings → Connectors → Advanced → Developer mode

https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check

Custom connectors are a paid-plan feature and take a remote URL only. ChatGPT cannot run a packaged server locally. ChatGPT docs

AI Crawler Checker — Robots.txt Policy for Six Bots in other clients