Site crawler
Crawl the whole site,not one URL.
Give Nevin a starting URL. It reads robots.txt and your sitemaps, follows internal links within a page and depth budget, and records what each URL actually returns.
What gets recorded
Every crawled URL keeps its own evidence, so a finding can always be traced back to what your server returned.
- HTTP status code and the final URL after redirects
- Each redirect hop, so chains and loops are visible
- Broken internal links, together with the pages that link to them
- Whether a page was found through a sitemap or through links
Bounded and polite by design
A crawl is a visitor, not a stress test. It stays inside the limits you and the plan set, and it asks permission first.
- Respects robots.txt and pauses between requests to the same host
- Stops at a page budget and a link-depth limit
- Re-checks every redirect hop and refuses private or internal network addresses
- Can be cancelled while it runs
From URL to findings
Enter a site, choose how many pages to cover, and follow progress while the crawl runs. Results appear as pages, issues and exports in your project.