Agent Crawl Readiness for LangChain
io.github.sadri-dridi/agent-crawl-readiness
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
client:LangChain
transport:streamable-http
tools:31
Install Agent Crawl Readiness in LangChain
pip install langchain-mcp-adapters
from langchain_mcp_adapters.client import MultiServerMCPClient
client = MultiServerMCPClient({
"agent-crawl-readiness": {
"transport": "streamable_http",
"url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp",
}
})
tools = await client.get_tools()
get_tools() returns the server's tools as LangChain tools, ready for an agent. LangChain docs
What LangChain can do once it is connected
- agent-crawl-readiness
- utc-time
- timezone
- validate-json
- normalize-url
- inspect-robots
- domain-shape
- citation
- compatibility
- status-catalog
- iana-zones
- web-fetch
- fetch-status
- github-repo-shape
- weather-hint
- memory-key-count
- file-path-ok
- browser-url-ok
- think-steps
- search-query-len
- hn-front-count
- lib-docs-hint
- playwright-url-ok
- figma-url-shape