Skip to content
mcp/harbor

Crawlwise llms.txt changelog

site.crawlwise/crawlwise

1 change recorded. MCP Harbor re-reads this listing from the official registry every six hours and asks the live server what it exposes, and keeps what changed between one look and the next.

  • llms.txt 1 field changed
    file
    absent published

    https://crawlwise.site/llms.txt 

    + # Crawlwise+ > SEO audit tool and AI crawler checker. Paste a URL and see what search engines and AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can actually read.+ Crawlwise audits a page for search discoverability, structured data, live crawler access and measured performance. Every finding shows the evidence it was based on: the HTML, the response header, the robots.txt rule or the live fetch. When something cannot be measured, the report says "unmeasured" instead of guessing a value. The free tools below need no account; full audits, site crawls, monitoring and the API use credits.+ ## Product+ - [Home](https://crawlwise.site/): Run an SEO and AI crawler audit on any public URL+ - [Plans](https://crawlwise.site/plans): Pricing, credits and what a full audit costs+ - [Methodology](https://crawlwise.site/methodology): What every check measures, how, and what it cannot measure+ - [Check catalogue (JSON)](https://crawlwise.site/methodology/checks.json): Machine-readable definition of every check+ - [Limits](https://crawlwise.site/limits): What an audit cannot see, what the checklist score is not, and why a metric can be missing+ ## Free tools+ - [Schema markup generator](https://crawlwise.site/tools/schema): Build JSON-LD for supported page types, with required-field checks, ready to paste into your head.+ - [Meta tag generator](https://crawlwise.site/tools/meta-tags): Title, description, robots directives, and canonical in one HTML snippet.+ - [robots.txt generator](https://crawlwise.site/tools/robots-txt): Write crawl rules for search and retrieval bots. Training preferences are a separate decision.+ - [Hreflang tag generator](https://crawlwise.site/tools/hreflang): Language and region alternates as HTML, HTTP headers, or XML sitemap markup.+ - [Redirect generator](https://crawlwise.site/tools/redirects): 301 and 302 rules for Apache, Nginx, Next.js, and Vercel, including bulk import.+ - [XML sitemap generator](https://crawlwise.site/tools/sitemap-generator): A valid urlset of canonical URLs with honest lastmod dates. Bulk URL support included.+ - [Disavow file generator](https://crawlwise.site/tools/disavow): Advanced maintenance for confirmed manual actions. Most sites never need this.+ - [Search preview](https://crawlwise.site/tools/serp): See how a title and description truncate in Google results on desktop and mobile. Approximate: Google may rewrite either one.+ - [Core Web Vitals checker](https://crawlwise.site/tools/vitals): Lab Lighthouse and field Chrome UX Report data from Google PageSpeed Insights.+ - [Content phrase inspector](https://crawlwise.site/tools/keyword-density): See which words and phrases your page actually repeats. There is no target percentage.+ - [Domain rating checker](https://crawlwise.site/tools/domain-rating): Look up Domain Rating for a host. An Ahrefs metric, not a Google ranking factor.+ - [SEO + generative search checklist](https://crawlwise.site/tools/seo-checklist): Work through what applies to your site. Items you mark not applicable are excluded, not counted against you.+ - [Generative search readiness review](https://crawlwise.site/tools/visibility): Inspect the source signals a retriever can see. No score: actual citations are not measured here.+ - [llms.txt generator](https://crawlwise.site/tools/llms-txt): Create a Markdown overview for tools that support this format. Experimental: it does not improve Google rankings or guarantee AI citations.+ - [AI crawler checker](https://crawlwise.site/tools/ai-crawler-check): See which AI crawlers can read your URL — ChatGPT, Claude, Perplexity and more. Find fixable blocks before you miss citations. Free, no sign-up.+ - [Schema / Rich Results validator](https://crawlwise.site/tools/schema-validator): Validate JSON-LD schema. A Rich Results Test alternative: eligible, ineligible, or not applicable. Free, no sign-up.+ - [Content question planner](https://crawlwise.site/tools/query-fanout): Draft the questions your page should answer. Editorial prompts, not engine queries.+ - [Broken link checker](https://crawlwise.site/tools/broken-link-checker): Check every link on one page and see which ones fail. Internal and outbound, status by status. Free, no sign-up.+ - [Redirect chain checker](https://crawlwise.site/tools/redirect-chain): Follow every hop a URL takes and see the chain, the loops, and the final destination. Free, no sign-up.+ - [Social and search preview](https://crawlwise.site/tools/meta-preview): One URL, previewed as Google, Facebook, X, LinkedIn, WhatsApp, and Slack would show it. Free, no sign-up.+ - [robots.txt tester](https://crawlwise.site/tools/robots-tester): Does this URL get blocked, for this crawler? Test the real file against a real path. Free, no sign-up.+ - [Sitemap validator](https://crawlwise.site/tools/sitemap-validator): Fetch a sitemap or sitemap index and check every URL entry: the XML, the limits, and where the URLs actually land. Free, no sign-up.+ - [Page speed comparison](https://crawlwise.site/tools/speed-compare): Two URLs, the same Lighthouse run each, side by side. Free, no sign-up.+ - [HTTP header inspector](https://crawlwise.site/tools/headers): Every response header for a URL, with the security and caching headers explained. Free, no sign-up.+ - [Mixed content checker](https://crawlwise.site/tools/mixed-content): Find HTTP resources loaded by an HTTPS page, which browsers block or warn on. Free, no sign-up.+ - [Canonical checker](https://crawlwise.site/tools/canonical-checker): Read the canonical a page declares, and what it points at. Free, no sign-up.+ - [Structured data extractor](https://crawlwise.site/tools/structured-data): Every JSON-LD entity on a page, nested and readable, with the types and properties as published. Free, no sign-up.+ - [Heading checker](https://crawlwise.site/tools/heading-checker): List every H1–H6 on a page in document order, with the H1 count, empty headings, and level jumps shown as observations, not errors. Free, no sign-up.+ - [HTTP status code checker](https://crawlwise.site/tools/http-status-checker): Check the HTTP status code of up to 10 URLs at once: the final code, every redirect hop with its Location, response time, and the headers that explain it. Free, no sign-up.+ ## Research

    Showing the first 40 of each.

Crawlwise MCP server