Runs on request only
There is no continuous crawl and no background queue. It runs only after someone enters an address in the form.
MrRobot-SEO-Analyser
Our crawler says who it is, and respects your rules.
MrRobot-SEO-Analyser is the crawler behind the SEO/GEO Analyser. It runs only when someone asks for a specific page to be checked, requests a small number of addresses, and does not collect content to train models.
User agent string
Mozilla/5.0 (compatible; MrRobot-SEO-Analyser/1.0; +https://mrrobot.co.il/bot)
At most 25 requests per check
One page, robots.txt, the sitemap, the canonical, language versions and the AI files - fewer than 25 requests in total, and no more than four at a time to the same server.
Read only
GET requests only. The crawler never submits forms, never signs in, and never changes anything on the site.
Respects robots.txt
Addresses other than the page you asked about are fetched only when robots.txt allows it for our crawler.
Allow the crawler
Nothing to do. A crawler that is not explicitly blocked is allowed by default.
Block the crawler
Add these lines to robots.txt at the site root. The check still runs, and reports openly that the page is blocked for our crawler.
User-agent: MrRobot-SEO-Analyser
Disallow: /A question about the crawler?
Get in touch any time and we will explain exactly what is checked and why.