FAIR Crawlers Policy
AI Crawlers and Robots Policy
An AI crawlers policy should explicitly describe access expectations, canonical evidence surfaces, and governance for indexing-related updates.
- Define crawler expectations and policy boundaries clearly.
- Cross-link policy with privacy, terms, and evidence index pages.
- Maintain clear effective dates and change logs for policy revisions.
- Should bot classes be named directly? Yes, when treatment differs.
- Does this replace robots.txt? No, it complements technical directives.
- What is the goal? Predictable indexing behavior and citation hygiene.