Glossary
robots.txt
A website file expressing crawling directives for user agents. Interpretation depends on the crawler and the specific directive.
In practice
A rule for one crawler does not describe every AI system’s access.
This is Outrigger’s working definition. Always check a study’s operational definition before comparing its findings with another study.
Read in context
Related term: Structured data.
How we measure AI visibility · Published study statistics · All glossary terms
See where your brand stands
Start with what AI says about your business.
See what AI says about youSee pricing