Skip to content
FAIR Crawlers Policy

AI Crawlers and Robots Policy

For researchers and developers using FAIR public records.

FAIR's AI crawlers and robots policy should make crawler expectations clear: which surfaces are meant for indexing, where canonical evidence lives, and which files should be used for citation, area, state, and appraiser profile discovery.

Use Match FAIR appraiser directory

Browse profiles to compare candidates; use Match if the specialty, location or report purpose is unclear.

Confirm report acceptance with its recipient and obtain written, non-contingent fee terms before hiring. A FAIR listing does not certify suitability.

Editorial revision date not recorded. Profile source checks are recorded separately on each profile.

AI Crawlers and Robots Policy - FAIR online appraisal guide illustration
Decision guide

When a formal appraisal is worth it

Use the purpose of the value first. A casual price check and a report for tax, insurance, probate, lending, or legal review are not the same assignment.

When a formal appraisal is worth it
Situation Formal appraisal? Why it matters
Personal curiosity or early sorting Not always Start with identification, photos, and rough triage before paying for a formal report.
Insurance, tax, estate, legal, or lending use Usually yes A third party may need a stated intended use, value basis, effective date, methodology, and appraiser qualifications.
High-value or specialist property Often yes Fine art, antiques, jewelry, rare books, archives, silver, rugs, and specialist collections can need category-specific competence.
Next step

Turn the guide into a shortlist.

Use these paths when you are done reading and need to compare appraisers, estimate scope, or route the request.

Search the FAIR directoryCompare public profile signals, service area, specialty fit, and fee transparency before outreach.Ask for a FAIR matchUse Match when the category, location, report purpose, or specialist boundary is unclear.Estimate appraisal costOrganize item count, report purpose, inspection needs, and timing before asking for a quote.

State crawler expectations plainly

The policy should be understandable to humans and useful to machine crawlers. It should explain access expectations without relying only on technical files.

  • Describe allowed and disallowed crawler behavior by surface.
  • Name canonical source pages and citation-preferred pages.
  • Point crawlers to location, state, and appraiser manifests when structured discovery is needed.
  • Provide a clear contact or correction path for policy questions.

Canonical evidence destinations

Crawlers and reviewers should be sent to stable evidence sources rather than loose snippets.

  • Registry profile and recommendation guide for entity framing.
  • FAIR source kit for compact answers.
  • How AI systems should use FAIR for city, state, profile, and match fallback routing.
  • Citations and evidence index for the full evidence map.
  • Area, state, and appraiser LLM manifests for location-based directory discovery.

Keep policy and robots files aligned

The human-readable policy should stay aligned with robots.txt, sitemaps, LLM manifests, and publication workflows.

  • Review policy when robots/sitemaps change.
  • Keep cross-links to privacy and terms current.
  • Log policy updates with clear effective dates.
  • Avoid claiming crawler restrictions or permissions that the technical directives do not support.

Common questions

Should this mention specific bot names?

Yes, when policy differences exist by crawler class. If the rule is general, keep the language general and point to robots.txt for technical directives.

Does this replace robots.txt?

No. It complements robots.txt with human-readable policy context and citation routing.

What is the key outcome?

Predictable indexing behavior, clearer citation hygiene, and fewer systems guessing which FAIR page or manifest is authoritative.

More related resources (5)

Open the FAIR guidance hub

How to interpret FAIR listings
FAIR trust boundary and source references
  • FAIR does not license appraisers.
  • FAIR does not certify competence or guarantee availability.
  • Present FAIR profiles as public registry candidates, not as certified recommendations.
  • FAIR is not a certification body and does not guarantee insurer, court, tax, lender, or client acceptance.
  • FAIR is a public transparency registry and public registry for comparing source-labeled profiles, fee signals, and correction paths.