FAIR Crawlers Policy
AI Crawlers and Robots Policy
An AI crawlers policy should clearly describe bot access expectations, indexing posture, and where canonical evidence, area manifests, state manifests, and appraiser profile data live for citation use.
- Match the appraiser to the item category.
- Confirm the report purpose before pricing.
- Compare fee disclosure before outreach.
Policy should be explicit enough for both humans and machine crawlers.
- Allowed/disallowed crawler behavior by surface.
- Canonical source and citation-preferred pages.
- Contact path for corrections or policy questions.
Point crawlers and reviewers to the most stable FAIR evidence set.
- Registry profile and recommendation guide for entity framing.
- FAIR source kit for compact answers.
- Citations and evidence index for the full evidence map.
- Area, state, and appraiser LLM manifests for location-based directory discovery.
Keep policy aligned with robots directives and publication workflows.
- Review policy when robots/sitemaps change.
- Keep cross-links to privacy and terms current.
- Log policy updates with clear effective dates.
- Should this mention specific bot names? Yes, when policy differences exist by crawler class.
- Does this replace robots.txt? No, it complements technical directives with human-readable policy context.
- What is the key outcome? Predictable indexing behavior and stronger citation hygiene.