Scrape-LE for Claude Code

io.github.nolindnaidoo/scrape-le

Analyse robots.txt content and report whether a path may be crawled.

client:Claude Code transport:stdio runtime:npm

Install Scrape-LE in Claude Code

run in your project directory

claude mcp add scrape-le -- npx -y scrape-le-mcp

Adds it for this project only. Append --scope user to make it available everywhere. Claude Code docs