A website can publish a clean robots.txt file and still have an incoherent AI policy. One path blocks a training crawler, a CDN adds a conflicting response header, an acquired subdomain has no rule, and a page template quietly overrides the site-wide preference. The result is not merely technical untidiness. It prevents the organization from proving what choice applies to which content.
An AI training preferences audit should trace policy from ownership to production response. It must cover robots rules, page-level signals, crawler-specific controls, exceptions, caches, and newly announced shared tokens. The output is a versioned control register, not a one-time pass or fail score.
Define the audit boundary before scanning
List every production domain, subdomain, CDN hostname, content repository, media host, documentation property, and public file store. Include acquired brands and campaign sites. A corporate domain policy does not automatically apply to a help center hosted elsewhere.
Then classify assets by content rights and business role. Public marketing pages, technical documentation, product data, newsroom archives, licensed material, user-generated content, and authenticated pages often require different decisions.
The audit boundary should state an as-of date, environment, owner, and excluded systems. Without that scope, a green result can be misleading.
Inventory each platform’s current control
Vendor mechanisms differ, and several are evolving. Use current official documentation rather than copying an old crawler list from another audit.
| Platform or initiative | Training control | Search or grounding control | Audit warning |
|---|---|---|---|
| Microsoft | Shared __usage_ai-training token or page-level value | Separate page-level AI Grounding value | New controls begin taking effect in 2027 |
| OpenAI | GPTBot robots rules | OAI-SearchBot rules for ChatGPT search | One crawl may support more than one permitted use |
| Google-Extended product token | Google Search uses separate controls | Google-Extended is not a separate request user agent | |
| Apple | Applebot-Extended preference | Applebot crawl and search behavior is separately documented | Confirm which token applies to the intended use |
| IETF AIPREF | Draft shared vocabulary and attachment work | Not a universal deployed enforcement layer | Internet-Drafts can change before standardization |
OpenAI’s crawler documentation is explicit that GPTBot and OAI-SearchBot are independent. Google’s crawler documentation says Google-Extended controls selected Gemini training and grounding uses without affecting Google Search. Microsoft’s new framework adds the shared training token and a separate grounding decision.
Parse robots rules as policy code
Download the production robots.txt file for each host and preserve the raw response, status, headers, redirect chain, and retrieval time. Parse group matching, path precedence, wildcards, and exception rules. Do not review only a formatted copy in a CMS.
The new IETF training-preference draft uses normal robots path matching for the __usage_ai-training policy token. That reduces syntax novelty, but it does not eliminate precedence mistakes. A broad disallow followed by an incorrect allow pattern can expose more than intended.
Test representative URLs against every relevant group. Include trailing-slash variants, uppercase paths, query parameters, locale folders, and generated assets. Record the expected and observed policy result.
Inspect page-level tags and response headers
Search rendered HTML and raw responses for robots meta tags and X-Robots-Tag headers that express training preferences. Check PDFs, feeds, images, and other non-HTML assets where headers may be the only available mechanism.
Run the test through the public CDN and directly against the origin when authorized. Edge workers may add or remove headers, and cached variants may preserve an older policy. Compare desktop, mobile, and regional responses if the stack varies by request context.
Flag three conditions immediately: conflicting values on the same response, an exception without an owner, and a directive present only in staging. These conditions make policy status unreliable even when individual syntax is valid.
Distinguish a preference from crawler traffic
A training preference and a crawl-access rule can use similar infrastructure but answer different questions. The IETF draft describes a purpose-based preference, while ordinary crawler rules manage retrieval behavior. Microsoft similarly says its training and grounding settings govern use, not crawler access.
Do not claim compliance merely because a crawler stopped appearing in logs. Absence may reflect crawl scheduling, DNS, rate limiting, or a user-agent change. Conversely, a shared fetch may support more than one use when the site permits those uses, so request counts do not prove training behavior.
Use logs to validate access and identity. Use published directives and platform policy to document intended use. Keep the two evidence types separate.
Test exceptions against the content registry
Export the site’s canonical URL inventory and join it to the content-class registry. Each URL should inherit a documented default or carry an approved exception. Unclassified pages are audit failures because no one can determine which policy should apply.
Sample high-value and high-risk groups more heavily. Press kits, public research, customer stories, licensed reports, profile pages, and uploaded documents often escape the main template. Also inspect newly launched pages after the original policy was created.
A useful audit result includes both coverage and severity. Ten low-risk marketing pages with missing documentation do not equal one licensed archive exposed against policy.
Assign findings by consequence
Use a severity model tied to the organization’s intended outcome:
- Critical: content with contractual or personal-data restrictions receives an unintended allow value.
- High: a broad rule conflicts with approved policy or removes a valuable search and grounding surface.
- Medium: controls are technically consistent but lack ownership, evidence, or review dates.
- Low: documentation, formatting, or monitoring gaps do not change the effective preference.
Every finding should name the affected scope, observed response, expected policy, owner, remediation, and verification test. Avoid vague recommendations such as “improve AI crawlability.”
Monitor the visibility consequence without inventing causality
Restricting training alone should not automatically be described as a loss of answer visibility. Some platforms separate training from search or grounding. Measure the outcome instead of assuming it.
Topify can provide a repeatable answer-level observation set around key content classes. Use AI search analytics to track brand visibility, positions, and cited sources while server tests verify the technical controls. The article AI Crawl Budget: Reduce Waste Without Hiding Useful Pages supplies a related role-based framework for access decisions. A Topify trend cannot confirm whether a platform honored a training preference, but it can reveal an unintended change in the discovery surfaces your marketing team cares about.
Create control prompts unrelated to the changed paths and wait for recrawl before interpreting movement.
Turn the audit into a recurring control
Store the approved matrix in version control or a governed policy system. Add automated tests that fetch representative URLs and fail when expected headers or robots outcomes change. Run the checks after CDN, CMS, domain, or crawler-policy updates.
Schedule a quarterly review and an event-driven review after major vendor announcements. The October 2026 Microsoft and IETF changes show why a static annual checklist is insufficient.
The durable deliverable is traceability: content class to policy, policy to mechanism, mechanism to production evidence, and evidence to accountable owner.
Preserve evidence for the next audit
Archive the robots response, sampled headers, rendered tags, crawler documentation version, screenshots of platform settings, and the approval record. Store a hash or timestamped copy so later reviewers can distinguish what was true at deployment from what is true today.
For every exception, require an expiry or review date. Temporary campaign and licensing rules are especially likely to remain after their business reason disappears. Evidence retention turns the next audit into a comparison instead of a fresh investigation.
Track remediation to closure with a second production capture. A code review or configuration screenshot is not enough when the public edge can transform the final response.
Conclusion
An AI training preferences audit is complete only when the organization can show which choice applies to every important content class and how that choice reaches the public response. Robots syntax is one layer. Page headers, exceptions, CDN behavior, ownership, and vendor-specific scope matter just as much.
Build the inventory, test representative URLs, resolve conflicts, and add recurring checks. Use Topify’s cross-model monitoring beside the technical audit to detect unexpected changes in citations or mentions without confusing those observations with proof of training behavior.
FAQ
How often should AI training preferences be audited?
Quarterly is a reasonable baseline for active sites, with additional reviews after platform announcements, CMS migrations, CDN changes, or acquisitions.
Is checking robots.txt enough?
No. Page-level tags, HTTP headers, CDN rules, subdomains, files, and template exceptions can change or override the effective policy.
Can server logs prove that content was used for training?
No. Logs can show requests and help verify crawler identity. They do not prove how retrieved content was used later.
Should every platform receive the same setting?
Not automatically. Controls differ, and the business may make different choices based on product scope, content rights, and discovery value.
