robots txt checker for LangChain
io.github.ryuamarines/robots-txt-checker
Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).
client:LangChain
transport:streamable-http
tools:1
Install robots txt checker in LangChain
pip install langchain-mcp-adapters
from langchain_mcp_adapters.client import MultiServerMCPClient
client = MultiServerMCPClient({
"robots-txt-checker": {
"transport": "streamable_http",
"url": "https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp",
}
})
tools = await client.get_tools()
get_tools() returns the server's tools as LangChain tools, ready for an agent. LangChain docs