Which AI crawlers can access this site?
A free Chrome extension. One click on any website shows you exactly which AI bots that site blocks or allows in its robots.txt.
Chrome Web Store listing is in review. Install link goes here once it is approved.
The popup, run against a news site. Most publishers block AI training bots and allow the assistant ones.
What it tells you
Click the icon and the extension fetches the site's robots.txt, works out which rules apply to each AI crawler, and gives you a verdict per bot.
Blocked Disallowed from the whole site.
Partial Allowed in, but some paths are closed off.
Allowed No restrictions apply.
It also checks whether the site publishes an llms.txt file, and whether the page you are on carries restrictive meta robots tags. Both matter, and neither shows up in a normal robots.txt check.
Why it exists
AI crawler rules have become their own layer of technical SEO, and they are invisible. Two sites can look identical in every audit tool and have completely opposite positions on whether their content feeds AI answers. Reading that by hand means opening robots.txt, finding the right user-agent group, and applying the longest-match rule correctly. This does it in a click.
Correct parsing
Follows RFC 9309. Longest match wins, ties go to Allow, a named group overrides the wildcard group. Most checkers just take the first matching line and get it wrong.
Page as well as site
Site-wide verdict plus the answer for the exact URL you are on, when the two differ.
20 crawlers
Training bots, AI search bots, and assistant user agents across the major AI companies, each labelled with what it is actually for.
Copy the report
One button takes the whole result away as plain text, ready to paste into an audit.
Who it is for
- SEOs auditing whether a client is visible to AI search, or accidentally invisible.
- Publishers deciding what to allow, and checking what competitors decided.
- Site owners who want to know what their robots.txt is actually doing, as opposed to what they think it says.
What it does not do
No accounts, no analytics, no tracking, no servers. It runs only when you click it, only on the tab you clicked, and it stores nothing. The full statement is in the privacy policy.