LinkSweep crawler specification

Seeing LinkSweep/1.0 in your logs? That's our crawler, checking a site that asked to be monitored. It comes from a single fixed IP with an identifiable user-agent, respects robots.txt, and rate-limits itself per host. Everything you need to identify or allowlist it is below.

Identity

Source IP: 185.31.243.206
User-Agent: Mozilla/5.0 (compatible; LinkSweep/1.0; +https://linksweep.eu/bot)
User-Agent (mobile checks): Mozilla/5.0 (Linux; Android 10) … Chrome/120.0.0.0 Mobile Safari/537.36 LinkSweep/1.0
Operator: LEAD OÜ, Tallinn, Estonia — see About

How it behaves

Respects robots.txt. It checks robots.txt and meta-robots rules before crawling and honours disallow, noindex and crawl-delay directives.
Rate-limits per host. It caps concurrent requests to a single origin and adds an adaptive delay that grows on 429/503 responses (honouring Retry-After) and relaxes once responses are clean.
Fetches only what it needs. Your pages, their linked resources, plus robots.txt and your XML sitemap — a few hundred requests per scan for a typical site.
Renders in a real browser when required. JavaScript sites are loaded in headless Chromium, self-hosted on our EU servers; your URLs aren't sent to any third-party service.

Allowlisting behind a firewall

If a web application firewall blocks the crawler, LinkSweep identifies which one and links you to a step-by-step guide. Allowlist the IP and user-agent above to let it through:

Cloudflare → Akamai → Imperva → AWS WAF → DataDome → PerimeterX → Sucuri → Fastly → Wordfence →

Questions about the crawler? Contact us.