AI Crawler Checker — Robots.txt Policy for Six Bots for LangChain
io.github.kriton-dev/ai-crawler-access-check
Check declared robots.txt permissions for six AI crawlers at URL paths
client:LangChain
transport:streamable-http
Install AI Crawler Checker — Robots.txt Policy for Six Bots in LangChain
pip install langchain-mcp-adapters
from langchain_mcp_adapters.client import MultiServerMCPClient
client = MultiServerMCPClient({
"ai-crawler-access-check": {
"transport": "streamable_http",
"url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
}
})
tools = await client.get_tools()
get_tools() returns the server's tools as LangChain tools, ready for an agent. LangChain docs