AI-crawler readiness
Can the crawlers you care aboutreach your pages?
Nevin checks whether robots.txt lets well-known search and AI crawlers fetch your pages, and whether the discovery files that help them find content are in place.
Per-crawler robots.txt access
A single rule can shut out a whole crawler. Nevin evaluates your robots.txt for well-known search and AI crawler user agents and reports which ones may fetch your pages.
- Allowed or blocked, per crawler user agent
- Unreachable or server-error robots.txt files, which crawlers may treat as a full block
Discovery signals
Sitemaps and an optional llms.txt file help crawlers find what you want them to read.
- Sitemaps referenced from robots.txt or found at the default location
- Sitemap files that do not parse, or list URLs that fail
- llms.txt, reported when present as an optional signal and not as a ranking factor
Readiness is not visibility
Nevin does not sample prompts, does not query ChatGPT, Gemini, Claude, Perplexity or any other AI engine, and does not track whether a model mentions or cites your brand. A page can be perfectly reachable and never be cited. Readiness tells you what is in your control: whether crawlers can get in.