Skip to content
mcp/harbor
LangChain

robotstxt ai for LangChain

io.github.sharozdawa/robotstxt-ai

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

client:LangChain transport:stdio runtime:npm

Install robotstxt ai in LangChain

pip install langchain-mcp-adapters 

from langchain_mcp_adapters.client import MultiServerMCPClient

client = MultiServerMCPClient({
        "robotstxt-ai": {
            "transport": "stdio",
            "command": "npx",
            "args": ["-y", "robotstxt-ai-mcp"],
        }
})

tools = await client.get_tools()

get_tools() returns the server's tools as LangChain tools, ready for an agent. LangChain docs

robotstxt ai in other clients

robotstxt ai MCP server