Can AI read your site?
Your robots.txt file can tell an AI crawler to stay out. We read it with your homepage and show what each of 9 crawlers is told.
What we read
- Your robots.txt. Which of 9 crawlers from OpenAI, Anthropic, Perplexity and Google it tells to stay out.
- Your homepage. The number of words in the page before any script runs.
- Two files AI agents ask for. Whether your site has an llms.txt file and a plain-text copy.
Your robots.txt tells Perplexity’s PerplexityBot to stay out of the page we checked.
- OAI-SearchBotAllowed
- Claude-SearchBotAllowed
- PerplexityBotTold to stay out
- GooglebotAllowed
A firewall can turn a crawler away even when robots.txt lets it in. This check does not test that. See our privacy policy.
Checking
Check another siteReading your site
We make four requests under our own name. It takes a few seconds.
- Your robots.txt
- Your homepage
- Your homepage as plain text
- Your llms.txt
What each crawler is told
The 9 AI crawlers we looked for in your robots.txt.
Crawlers that read pages for answers
Crawlers that collect training data Keeping these out is your choice.
What your homepage sends
What the page holds before any script runs.
Words
Two optional files
- Plain-text copy
- llms.txt
Next
See who ChatGPT mentions when your buyers ask.
We ask ChatGPT 10 questions your buyers would ask and show who each answer mentions. It takes about a minute. Your website is already filled in.
Run the checkWhat this check leaves out
A firewall can turn a crawler away even when robots.txt lets it in. This check does not test that. The free snapshot requests your pages under the names of six AI crawlers and reports any refusal it can confirm.
Get the free snapshot