Free Tool
AI Crawler Access Checker
Many sites block AI crawlers by accident — which means ChatGPT, Claude, and Perplexity can never learn about them. Enter a URL to see who is allowed in, and whether you have an llms.txt.
Video transcript
Is your site invisible to ChatGPT? It could be one small file. Your robots file tells crawlers what they may read. Many sites block AI crawlers by accident. A template file, a security plugin or a block bots setting can do it. Step one. Open the free AI Crawler Access Checker. Step two. Type your domain and press Check site. Step three. Read the result. Ten AI crawlers, from GPTBot to Bytespider, each allowed or blocked. Some publishers block on purpose. The New York Times site blocks all ten. An open site allows all ten. To fix it, add User-agent and Allow rules to your robots file. Then publish an llms.txt file as a clean summary. Check your site for free at mustardseedmt.com.
Why would you block an AI crawler?
Some publishers block AI crawlers on purpose — to protect content from being used in training. But many small businesses block them by accident: a template robots.txt, a security plugin, or a "block bots" setting can quietly shut out GPTBot and Google-Extended. If you want to be recommended by AI tools, that is exactly the wrong outcome.
To allow them, add explicit User-agent / Allow rules to your robots.txt, and publish an llms.txt so AI tools have a clean summary to work from. It is a core part of AI visibility (AEO/GEO).
Curious how common this actually is? We checked robots.txt on 1,000 real websites — see the full study.