crawler

DeltaVBot

DeltaVBot is the crawler behind search.deltavdevs.com, a developer-facing search engine run by DeltaVDevs.

How to recognise it

User agent: DeltaVBot/1.0 (+https://search.deltavdevs.com/bot)

How it behaves

Reads robots.txt (RFC 9309) before fetching, including Crawl-delay (capped at 30s). A robots.txt that errors with 5xx is treated as “do not crawl”.
One request per second per host by default, with backoff on 429/503 and Retry-After respected. Failing hosts are quarantined with exponential backoff.
Only fetches text/html pages up to 2 MB, and uses conditional requests (If-None-Match, If-Modified-Since) when it revisits.
Honors noindex, nofollow and noarchive (with noarchive we keep metadata only, never the text).
Seeds come from the external links cited by Wikipedia (read from the official dumps, not by crawling Wikipedia), plus DeltaVDevs’ own sites.

What we keep

Search keeps a versioned record of each page (title, metadata, structured data, links and a compressed copy of the main text) so you can query how the web changed. See the API.

Opting out

Block DeltaVBot in robots.txt to stop crawling, or use the opt-out form to also have stored data purged. Questions: search@deltavdevs.com.