Support and documentation for the AI Crawler Check browser extension. If something is not working or you have a question, the answer is probably below.
Click the extension icon on any website and it reads that site's robots.txt file, then shows you whether each known AI crawler user agent is allowed or blocked. It also checks whether the site publishes an llms.txt file, and whether the page you are on carries restrictive meta robots tags.
It covers 20 AI crawler user agents across OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Amazon, ByteDance, Common Crawl, Cohere, Mistral, DuckDuckGo, the Allen Institute, and Diffbot.
Nothing runs until you click the icon, and it only ever looks at the tab you clicked it on.
Blocked The bot is disallowed from the entire site with Disallow: /.
Partial The bot can reach the site, but some paths are disallowed.
Allowed No restrictions apply to that bot.
The chip shows the site-wide verdict. The smaller line underneath tells you whether the specific page you are on is blocked or allowed, in the cases where that differs from the site-wide answer.
Parsing follows RFC 9309: the longest matching rule wins, ties resolve in favour of Allow, and a named user-agent group overrides the * group.
There are no other permissions, no background activity, no accounts, no analytics, and no servers. See the privacy policy for the full statement.
If a site redirects its robots.txt to a different hostname, for example from the apex domain to www, the fetch can fail. The activeTab permission only grants access to the domain you are actually on. Open the site's main domain directly and run the check again from there.
Then no crawler is restricted by it, and every bot reads as allowed. That is the correct answer, not an error.
Most robots.txt checkers ignore the longest-match rule and just take the first matching line. This extension follows the RFC. Where the two disagree, check the actual file and count the path lengths.
The extension does not run on Chrome's internal pages such as chrome://extensions, the Web Store, or local files. Move to a normal http or https page.
Questions, bugs, or a site that produces a wrong verdict: email joshmarwords@gmail.com. If you are reporting a wrong verdict, include the URL so the robots.txt can be checked directly.