robots txt checker for Claude Code

io.github.ryuamarines/robots-txt-checker

Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).

client:Claude Code transport:streamable-http tools:1

Install robots txt checker in Claude Code

run in your project directory

claude mcp add --transport http robots-txt-checker https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp

Adds it for this project only. Append --scope user to make it available everywhere. Claude Code docs