Robots controls access, not citations
Allowing crawler access only means a page can be requested; it does not guarantee indexing or citation. Blocking access can remove public content from many AI retrieval paths.
Technical guide
AI Crawlers and robots.txt
The first step in AI visibility is making sure important public pages are not accidentally blocked by robots, noindex, auth walls, or client-only rendering.
Check crawler accessnoindex is a separate gate
robots.txt usually controls whether crawlers may request a path. A page-level noindex directive controls whether it should be indexed. Reports, dashboards, and private pages should stay noindex; public resources should not.
Rendering is also an access issue
If important copy only appears after client-side JavaScript runs, some crawlers may miss it. Definitions, FAQ, pricing, policies, and product explanations should be present in server-rendered HTML.
Core checks
Whether robots.txt allows important public pages
Whether public content pages avoid noindex
Whether API, report, admin, and temporary results avoid indexing
Whether core content appears in initial HTML
Whether sitemap lists discoverable resource pages
FAQ
Should I allow every AI crawler?
It depends on your content strategy. Public content intended for AI search exposure usually should be crawlable; private, paid, or sensitive content needs stricter rules.
Does llms.txt replace robots.txt?
No. robots.txt is an access convention. llms.txt is closer to a content index for models and tools. They can complement each other, but they do not replace each other.