polite-crawler for Grok
io.github.mark1russell7/polite-crawler
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
client:Grok
transport:stdio
runtime:npm
Install polite-crawler in Grok
.mcp.json (in your project root)
{
"mcpServers": {
"polite-crawler": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"polite-crawler"
],
"env": {
"POLITE_CRAWLER_DIR": "<POLITE_CRAWLER_DIR>",
"POLITE_CRAWLER_CONTACT": "<POLITE_CRAWLER_CONTACT>",
"POLITE_CRAWLER_ALLOW_PRIVATE": "<POLITE_CRAWLER_ALLOW_PRIVATE>",
"POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<POLITE_CRAWLER_ALLOW_USER_PROVIDED>"
}
}
}
}Grok Build reads Claude Code's .mcp.json and Claude Desktop's config unchanged, so a server you have already set up there needs nothing new. The Grok web app and X integration do not take custom servers. Grok docs