About CitableReportBot

CitableReportBot is the crawler that CitableReport uses to audit a website when someone asks for an AI citation report on it. It visits only when a person enters that site's address, reads a small number of public pages once, and does not come back unless asked again.

A crawler is a program that fetches web pages automatically. An AI citation report is an audit of how easily AI assistants can find, read, trust and quote a website; to produce one we have to read the site, which is what CitableReportBot does. A typical visit is about 30 requests over 10 to 30 seconds.

How does it identify itself?

It identifies itself with this user agent on every request:

CitableReportBot/1.0 (+https://citablereport.com/bot)

What does it fetch?

For each report it fetches a small, fixed set of things:

• /robots.txt, your XML sitemap and /llms.txt
• Your front page and up to 19 more public pages, at most four at a time
• Up to two author pages linked from article bylines
• Your front page twice more, once identifying as an AI crawler and once as a browser, to see whether a firewall treats them differently

It reads HTML only, does not run JavaScript, does not submit forms, does not sign in, and stops reading any page after 2 MB.

Does it obey robots.txt?

Yes. CitableReportBot obeys the robots.txt rules addressed to CitableReportBot, or to * if it is not named. If it is disallowed from your front page it stops, and the person who asked is told the site could not be audited.

How do I block it?

To keep CitableReportBot off your site, add this to your robots.txt:

User-agent: CitableReportBot
Disallow: /

How do I allow it through a firewall?

If you asked for a report and were told your site refused our crawler, your firewall or CDN is turning it away. Allow requests whose user agent contains CitableReportBot, then scan again. It is worth checking at the same time whether the same rule is turning AI crawlers away: see our AI crawlers list.

What happens to what it reads?

The pages are analysed and discarded. We keep measurements (word counts, which checks passed, short quotes used as evidence in the report), not copies of your pages. The content is not used to train any model. More detail is in our privacy policy.

How do I report a problem?

If CitableReportBot is misbehaving on your site, write to team@citablereport.com with your domain and we will look into it.