About Our Crawler
When someone asks us to audit a site, we fetch pages from it. Our crawler identifies itself honestly:
SiteCandorBot/1.0 (+https://sitecandor.com/bot)
How the SiteCandor Crawler Behaves
- It obeys your robots.txt — every rule in it, including the ones aimed at all robots. The single exception is your own site, read for you once you have proved it is yours. A rule that names
SiteCandorBotis how you decide about us in particular: always, everywhere, whoever asked, with nothing behind it. - It stays on the site it was asked about — that domain and its subdomains. A page that redirects somewhere else is not followed, and that other site is never fetched.
- It requests at most three pages a second from any one site, and one a second if the site pushes back.
- It reads pages. It never submits forms, logs in, or changes anything.
- An audit reads up to 250 pages of a site.
Your robots.txt Decides, Unless the Site Is Ours to Read
A rule like User-agent: * is aimed at every robot, and we are one, so it stops us. Somebody asking us for an audit does not change that: anyone can type any address into our form, and we have no way of knowing whose site it is. A stranger’s request is not your permission, so we do not treat it as if it were.
The exception is the site’s own people, and we act on it only when control is proved rather than claimed. A subscriber who connects their site’s Google Search Console has shown us something only the site can grant: Google lists them as an owner or a full user of it, which the site’s owner had to give them. For that site, and no other, a rule addressed to all robots is not addressed to us — the person who wrote it is the person asking. Everyone else, on every other site, gets the paragraph above. And a rule naming SiteCandorBot still stops us, proof or no proof.
A page we are kept out of is still reported. The audit says that page is hidden from search engines, because that is usually the entire problem the person came to us with: a rule left over from a staging site, quietly keeping a page out of Google for months. What we will not do is read the page behind the rule in order to say it.
Some of what we read, nobody asked us for at all. The competitor page analysis a subscriber runs against a page they compare with theirs is your robots.txt and that one page, roughly two requests. And from time to time we test our own crawler against real sites, because a crawler tested only against sites we built is one we cannot honestly vouch for: that reads more of a site, at the same three requests a second. Both obey your robots.txt exactly as an audit does, and naming SiteCandorBot stops both.
How to Block the SiteCandor Crawler
Copy these two lines into your robots.txt and we’ll stop — every product, every time, whoever asks:
User-agent: SiteCandorBot Disallow: /
To keep us off one part of the site and nothing else, name the path instead:
User-agent: SiteCandorBot Disallow: /members/
A rule aimed at all robots stops us too. These two lines are for when you want to keep us out while letting other robots in, or when you want the block to be unmistakable. No hard feelings either way, and no need to tell us why. We re-read robots.txt at the start of every check, so a block takes effect the next time someone asks for an audit rather than at some point later.
How to Let the SiteCandor Crawler Past Your robots.txt
If your robots.txt tells robots to stay out and you want an audit of your own site anyway, give us a rule of our own. A rule naming SiteCandorBot replaces the general one for us, and leaves every other robot exactly as it was:
User-agent: SiteCandorBot Allow: /
Put it anywhere in the file, then run the audit again — we read robots.txt at the start of every check, so there is nothing else to switch on. Take the two lines out when you are done and we go back to obeying your general rule.
Subscribers have a second way, which touches nothing on the site: connect the site’s Google Search Console from the dashboard. Google only grants that standing to the site’s owner or to someone they gave full access to, so it proves what the rule above asserts, and from then on we read that site past rules aimed at all robots. It changes nothing for any other robot, and nothing for any other site.
Why Our Crawler Is Visiting Your Site
Someone typed your address into our form and asked for an audit. Unless they have proved the site is theirs, we have no way to check that they own it. That is why the crawler obeys your robots.txt rather than treating their request as your permission, and why it is built to be harmless either way. It reads public pages one at a time, at most three a second, and slows down when your server pushes back. It does nothing a browser with JavaScript turned off could not do.
We do not sell the data, publish it, or use one person’s audit to build a directory of anybody else’s site. A report exists at an unguessable link for the person who asked for it.
What Our Crawler Fetches From Your Site
- Your pages, as HTML — the same bytes a search engine would receive.
robots.txt,sitemap.xmlandllms.txt, if you have them.- A
HEADrequest per link, to see whether it is broken, and per image, to see how large it is. Those are not downloads.
The Load Our Crawler Puts on Your Server
We ask for at most three pages a second per host, one at a time. If your server pushes back with a rate limit, a busy signal or a timeout, we slow to one a second for the rest of the audit. If your robots.txt sets a Crawl-delay longer than that, we use yours instead — we will slow down on request, but we will not speed up. A full 250-page audit therefore usually takes three to eight minutes, and never has more than one request of ours in flight.
If your server answers with a rate limit — a 429, or a 503 asking us to wait — we stop asking that host and wait: ten seconds, then twenty, then thirty, then two minutes if it keeps happening. When the answer names a wait of its own we take that instead, up to two minutes; anything longer and we leave that page unread rather than sit on your doorstep. One page behind a limiter costs that page. A site still refusing us on more than one ends the audit — we tell the person who asked that we could not read their site, which is more use to them than a report with most of it missing.
How to Let the SiteCandor Crawler Through
If a firewall, security setting or rate limit stopped an audit of your own site, allow requests whose user agent contains SiteCandorBot. Match the user agent rather than an IP address: our requests come from cloud servers whose addresses change. It reads one page at a time, at most three a second, so letting it through will not strain your server. Then run the audit again.
How to Reach Us About the Crawler
If our crawler has caused you a problem, or you think it has behaved badly, tell us through the contact form and a person will read it. We would genuinely rather hear about it than not.
Questions
Questions From Site Owners
Why Is SiteCandorBot Crawling My Site?
Somebody asked for an audit of it — or, now and then, we are testing our crawler against real sites rather than only against our own. Unless that somebody has proved the site is theirs, we cannot check whose site it is, so the crawler obeys your robots.txt and is built to be harmless either way. It reads public pages one at a time, at most three a second, and slows down when your server pushes back. It changes nothing.
How Do I Block It?
Name SiteCandorBot in your robots.txt and disallow it. That stops us everywhere, always, whoever asked, and whatever anybody has proved to us. A rule aimed at all robots stops us too — unless the person asking has proved the site is theirs, which is the one case we read past it. Naming us is the rule that has no exception behind it. We read robots.txt at the start of every check, so the block works the next time anyone asks for an audit. You do not need to tell us why.
I Have Disallowed All Robots. Will You Still Read My Pages?
Not unless the site's own owner asked us to. A rule aimed at every robot is aimed at us as well, and we obey it: anyone can type any address into our form, and a stranger's request is not your permission. The exception is the site's own people, proved rather than claimed — a subscriber who connects the site's Google Search Console, where Google itself lists them as an owner or a full user of it. Then a rule aimed at all robots is not aimed at us, because the person who wrote it is the person asking. A rule naming SiteCandorBot stops us in either case, and adding one that allows us is the other way to let us in.
Does It Obey My robots.txt, Then?
Yes, with one exception, and it is your own. We read the file at the start of every check and follow every rule in it — the one case we read past a rule aimed at all robots is a site whose owner has proved to us it is theirs, and even then a rule naming SiteCandorBot stops us. We use your Crawl-delay if you set one, and we back off when your server asks us to. When a rule keeps us off a page, the report still says that page is hidden from search engines — that is usually the whole problem the person was looking for — but we do not read the page in order to say it.
How Much Load Does It Put on My Server?
We ask for at most three pages a second, and never more than one at a time. If your server pushes back with a rate limit, a busy signal or a timeout, we slow to one a second for the rest of the audit. If your robots.txt sets a longer Crawl-delay, we use yours. A full 250-page audit usually takes three to eight minutes.
What Happens When My Rate Limiter Pushes Back?
We wait, and then we stop — we never keep knocking. The first rate limit pauses us on that host for ten seconds, the next for twenty, then thirty, then two minutes. If your answer asks for a particular wait we take yours instead, up to two minutes. One page still behind the limit after that is left unread, and the report says how much of the site we saw. If more than one page is, the audit ends and the person who asked is told their site would not let us read it, rather than being handed a report with most of their site missing.
Does It Log in, Submit Forms, or Click Buttons?
No. It fetches pages and reads them. It never submits a form, never authenticates, and never changes anything on your site.