About Our Crawler

When someone asks us to audit a site, we fetch pages from it. Our crawler identifies itself honestly:

SiteCandorBot/1.0 (+https://sitecandor.com/bot)

How the SiteCandor Crawler Behaves

• It obeys your robots.txt — every rule in it, including the ones aimed at all robots. The single exception is the address you have proved is yours, read for you past rules aimed at all robots: that address and its www, never a subdomain and never any other site. A rule that names SiteCandorBot is how you decide about us in particular: always, everywhere, whoever asked, with nothing behind it.
• If your server answers our request for robots.txt with a server error, a rate limit or no answer in time, we read nothing else from that host, and the report says your server failed. Search engines treat a robots.txt that fails this way as a reason to stop reading the site. A missing robots.txt (a 404 or similar) means no rules, and we read the site as normal.
• It reads only the site it was asked about — that address and its subdomains, never a domain above it. A page that redirects somewhere else is not followed. Elsewhere it only checks that a page or image the site links to loads, after reading that site’s robots.txt, and only when the file allows it.
• It requests at most three pages a second from any one host — each subdomain counts as its own — and one a second if the host pushes back.
• It reads pages. It never submits forms, logs in, or changes anything.
• A free audit reads up to 500 pages of a site, and a subscriber’s monthly audit up to 1,000.

Your robots.txt Decides, Unless the Site Is Ours to Read

A rule like User-agent: * is aimed at every robot, and we are one, so it stops us. Somebody asking us for an audit does not change that: anyone can type any address into our form, and we have no way of knowing whose site it is. A stranger’s request is not your permission, so we do not treat it as if it were.

The exception is the site’s own people, and we act on it only when control is proved rather than claimed. A subscriber who connects their site’s Google Search Console has shown us something only the site can grant: Google lists them as an owner or a full user of it, which the site’s owner had to give them. For that address, and no other, a rule addressed to all robots is not addressed to us — the person who wrote it is the person asking. A subdomain’s own robots.txt, such as a shop or a blog run on it, still binds us in full, and so does every other site. We ask Google again every day, and the exception lasts only while Google keeps confirming it. Everyone else, on every other site, gets the paragraph above. And a rule naming SiteCandorBot still stops us, proof or no proof.

A page we are kept out of is still reported. The audit says that page is hidden from search engines, because that is usually the entire problem the person came to us with: a rule left over from a staging site, quietly keeping a page out of Google for months. What we never do is read the page behind the rule in order to say it.

Some of what we read, nobody asked us for at all. The competitor page analysis a Pro subscriber runs against a page they compare with theirs reads your robots.txt afresh each time, then that one page and your llms.txt: about three requests, and none at all when another analysis has already read that page the same day. And from time to time we test our own crawler against real sites, because a crawler tested only against sites we built is one we cannot honestly vouch for: that reads more of a site, at the same three requests a second. Both obey your robots.txt exactly as an audit does, and naming SiteCandorBot stops both.

How to Block the SiteCandor Crawler

Copy these two lines into your robots.txt and we stop reading your pages — every product, every time, whoever asks. The one request that still reaches you is our read of the robots.txt itself, which is how we see the block:

User-agent: SiteCandorBot
Disallow: /

To keep us off one part of the site and nothing else, name the path instead:

User-agent: SiteCandorBot
Disallow: /members/

A rule aimed at all robots stops us too. These two lines are for when you want to keep us out while letting other robots in, or when you want the block to be unmistakable. No hard feelings either way, and no need to tell us why. We read robots.txt before every check that asks your site for anything, so a block takes effect the next time someone asks for an audit rather than at some point later.

How to Let the SiteCandor Crawler Past Your robots.txt

If your robots.txt tells robots to stay out and you want an audit of your own site anyway, give us a rule of our own. A rule naming SiteCandorBot replaces the general one for us, and leaves every other robot exactly as it was:

User-agent: SiteCandorBot
Allow: /

Put it anywhere in the file, then run the audit again — we read robots.txt at the start of every check, so there is nothing else to switch on. Take the two lines out when you are done and we go back to obeying your general rule.

Subscribers have a second way, which touches nothing on the site: connect the site’s Google Search Console from the dashboard. Google only grants that standing to the site’s owner or to someone they gave full access to, so it proves what the rule above asserts, and from then on we read that address past rules aimed at all robots, for as long as Google keeps confirming it. It changes nothing for any other robot, and nothing for its subdomains or any other site.

Why Our Crawler Is Visiting Your Site

Someone typed your address into our form and asked for an audit. Unless they have proved the site is theirs, we have no way to check that they own it. That is why the crawler obeys your robots.txt rather than treating their request as your permission, and why it is built to be harmless either way. It reads public pages one at a time, at most three a second, and slows down when your server pushes back. It does nothing a browser with JavaScript turned off could not do.

We do not sell the data, publish it, or use one person’s audit to build a directory of anybody else’s site. A report exists at an unguessable link for the person who asked for it.

What Our Crawler Fetches From Your Site

• It reads your pages as HTML, the same bytes a search engine would receive.
• It reads robots.txt, sitemap.xml and llms.txt, if you have them.
• It asks for /favicon.ico, the tab icon browsers look for, to see whether your site has one.
• It asks for one made-up address, /sitecandor-check-page-not-found, to see what your server sends for a page that doesn’t exist.
• It asks for your home page at each way of typing your address, with and without www and over http and https, to see whether they all lead to the same place.
• It sends a HEAD request for each link, to see whether the link is broken, and for each image, to see how large the image is. It sends a GET when the HEAD fails, and, at most 25 times in an audit, when your server sends the same page for addresses it doesn’t have, to compare a link’s page with that one. It checks only links and images whose own site’s robots.txt lets it read them.

The Load Our Crawler Puts on Your Server

We ask for at most three pages a second per host, one at a time. If your server pushes back with a rate limit, a busy signal or a timeout, we slow to one a second for the rest of the audit. If your robots.txt sets a Crawl-delay longer than that, we use yours instead — we slow down on request, but we never speed up. A 300-page site therefore takes about five minutes, a 1,000-page audit about twenty, and none ever has more than one request of ours in flight.

If your server answers with a rate limit — a 429, or a 503 asking us to wait — we stop asking that host and wait: ten seconds, then twenty, then thirty, then two minutes if it keeps happening. When the answer names a wait of its own we take that instead, up to two minutes; anything longer and we leave that page unread rather than sit on your doorstep. One page behind a limiter costs that page. A site still refusing us on more than one ends the audit — we tell the person who asked that we could not read their site, which is more use to them than a report with most of it missing.

If a firewall that has been letting us read starts turning us away partway through, we leave your site alone for five minutes, then carry on at one page every three seconds. If it still turns us away, the audit ends there.

How to Let the SiteCandor Crawler Through

If a firewall, security setting or rate limit stopped an audit of your own site, allow requests whose user agent contains SiteCandorBot. Match the user agent rather than an IP address: our requests come from cloud servers whose addresses change. It reads one page at a time, at most three a second, so letting it through will not strain your server. Then run the audit again.

How to Reach Us About the Crawler

If our crawler has caused you a problem, or you think it has behaved badly, tell us through the contact form and a person reads it. We would genuinely rather hear about it than not.

If a report is about your site and you think a finding in it is wrong, tell us through the contact form.