robots.txt for all important bots
For each bot: may it read your homepage, are only some areas blocked, or is it excluded entirely? Grouped by AI search, user-requested fetches, AI training and classic web search.
Free AI crawler check
May ChatGPT, Claude & Perplexity read your website?
CiteBoost evaluates your robots.txt for 47 AI and search bots, fetches your homepage with the real user agents of GPTBot, ClaudeBot and PerplexityBot and checks whether an llms.txt exists. No sign-up, done in seconds.
What is checked
Real access test
Many firewalls and CDNs block AI bots even when the robots.txt allows them. So we really fetch your homepage with the user agents of GPTBot, ClaudeBot and PerplexityBot.
llms.txt
Whether your website offers an llms.txt – a short overview of your most important pages for AI systems.
Text without JavaScript
AI crawlers usually do not run JavaScript. We count how much text your homepage delivers without it.
Frequently asked questions
Why does this matter?
ChatGPT, Claude, Perplexity and other AI assistants can only cite and recommend content their crawlers are allowed to read. A single line in the robots.txt or a firewall rule can make your website invisible to them.
What is the difference between AI search and AI training?
Search bots such as OAI-SearchBot or PerplexityBot fetch content for answers with sources – block them and you will not be cited there. Training bots such as GPTBot or Google-Extended collect text to train AI models. Many websites allow search and block training.
What does “partly blocked” mean?
The robots.txt has its own rules for this bot that block some areas – the homepage is allowed. Rules that apply to all bots (for example /wp-admin) do not count; we show them separately.
What happens to my data?
We keep the result for 5 days so you can open and share the report. After that it is deleted automatically – including an email address you sent it to. Only if you explicitly confirm tips & news do we keep your address until you unsubscribe.
Do you check my whole website?
The free check looks at the homepage and the robots.txt and llms.txt files. In the app you check every single page with and without JavaScript and see which AI bots actually visit your website.