polite-crawler for Claude Code
io.github.mark1russell7/polite-crawler
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
client:Claude Code
transport:stdio
runtime:npm
Install polite-crawler in Claude Code
run in your project directory
claude mcp add polite-crawler -e POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR> -e POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT> -e POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE> -e POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED> -- npx -y polite-crawlerAdds it for this project only. Append --scope user to make it available everywhere. Claude Code docs