Skip to content
mcp/harbor
Verified official

robots txt checker MCP

v1.0.93057

io.github.ryuamarines/robots-txt-checker

Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).

transport:streamable-http

Target client

run in your project directory 

claude mcp add --transport http robots-txt-checker https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp

Adds it for this project only. Append --scope user to make it available everywhere. Claude Code docs

This listing does not declare its tools. Connect the server and your client will discover them on the handshake.