Skip to content
mcp/harbor
Verified official

polite-crawler MCP

v0.1.0

io.github.mark1russell7/polite-crawler

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

transport:stdio runtime:npm
View source Website

Target client

run in your project directory

Held in this page only — never stored, logged, or sent anywhere but back to your screen. 

claude mcp add polite-crawler -e POLITE_CRAWLER_DIR=<POLITE_CRAWLER_DIR> -e POLITE_CRAWLER_CONTACT=<POLITE_CRAWLER_CONTACT> -e POLITE_CRAWLER_ALLOW_PRIVATE=<POLITE_CRAWLER_ALLOW_PRIVATE> -e POLITE_CRAWLER_ALLOW_USER_PROVIDED=<POLITE_CRAWLER_ALLOW_USER_PROVIDED> -- npx -y polite-crawler

setup guides: Claude Code Claude Desktop Cursor VS Code Zed Windsurf Cline Gemini CLI Grok LangChain

Adds it for this project only. Append --scope user to make it available everywhere. Claude Code docs

This listing does not declare its tools. Connect the server and your client will discover them on the handshake.