Nevin crawler

NevinBot

NevinBot is the crawler behind Nevin. If you found it in your server logs, a Nevin customer asked it to audit your site for technical SEO issues.

User agent

NevinBot/1.0 (+https://nevin.w3dev.app/bot). The robots.txt product token is NevinBot.

Who runs it

Customers of Nevin start crawls of their own sites, or of sites they are authorized to audit, from their account.

robots.txt

NevinBot reads /robots.txt before crawling, follows the rules for NevinBot (or for * when none exist) and honours Crawl-delay up to 5 seconds. If robots.txt returns a server error or cannot be reached, it does not crawl.

Crawl rate

Requests to a single host are spaced at least 250 ms apart, and a Crawl-delay of up to 5 seconds is respected. A crawl visits at most 500 pages.

What it fetches

Public HTML pages, robots.txt, sitemaps and llms.txt on the crawled site. It does not log in, submit forms, run JavaScript or request API endpoints and file downloads.

Block or slow NevinBot

User-agent: NevinBot
Disallow: /

# or slow it down (up to 5 seconds is honoured)
User-agent: NevinBot
Crawl-delay: 5

Questions or abuse reports? Use the contact page and mention the time and IP address from your logs.