This tutorial turns “AI indexing” into a practical playbook: what the main AI crawlers are, how to control access, how to earn citations, how to monitor and enforce, and how to build content that AI systems prefer to quote.

Start here if you run a small business

Most of this article goes deep. If you run a small business with a WordPress site and no dedicated dev team, here is the short version, three things that matter, and the rest you can skip or come back to later.

1. Decide whether you want AI training bots on your site, then say so in robots.txt. For most local service businesses the answer is: allow everything. You want ChatGPT and Perplexity to know your business exists and recommend you. Blocking training crawlers makes sense if you publish original research, proprietary data, or paid content. If that is not you, leave them allowed and move on. Part 3 below has the exact robots.txt patterns either way.
2. Structure your pages so AI can quote them. This is where the real opportunity is. AI assistants preferentially cite pages that answer a question directly in the first two or three sentences, use clear headings, and have clean schema markup. That is the same work that helps traditional SEO, so you are not doing anything extra. Part 6 covers the specifics.
3. Check once a quarter whether you are getting cited. Ask ChatGPT, Perplexity, and Google’s AI Overview the questions your customers ask (“best web designer in Salt Lake City,” “how much does a website cost in Utah”) and see whether you come up. Ten minutes, four times a year. That is your entire monitoring program at this scale.

Everything below expands on those three, plus enforcement, governance, and measurement for teams that need it. Parts 4, 9, 10, and 11 assume a dev team, a CDN, and log infrastructure, skip them if that is not your situation.

Part 1. What AI indexing actually is

Classic indexing means a search engine crawls your pages, stores representations, and retrieves them for results. AI indexing adds two paths:

1. Training ingestion
   Your page is used as training data for a model or for retrieval corpora. This is usually governed by crawler user agents and publisher control signals.
2. On-demand retrieval
   A model answers a prompt and fetches live pages to ground its response. If your page is accessible and clearly answers the question, it can be cited or paraphrased.

You need a policy for both. You might allow normal search and on-demand citation but block training ingestion. Or you might allow everything because the exposure helps you. Decide by business model, compliance needs, and risk tolerance.

Part 2. The crawlers and how to speak to them

Below are the most common actors you will see in logs. Vendor names and agents evolve, so recheck quarterly.

• Google Search and AI features
  Control with standard Googlebot rules for crawling and indexing. For model training controls outside Search, use the Google-Extended agent in robots.txt.
• OpenAI
  Controls via GPTBot in robots.txt. Some enterprise offerings can also respect site-level allowlists and denialists.
• Anthropic
  Controls via a public bot name such as ClaudeBot and may also use fetchers through hosting providers that identify themselves in headers.
• Common Crawl
  Controls via CCBot in robots.txt.
• Perplexity and other real-time answer engines
  Look for user agents documented in their policies, plus fetchers that present a browser-like UA. Treat with strict allowlists until you verify compliance.
• Other ecosystems
  Vertical assistants, academic crawlers, and corporate LLMs often identify themselves in UA or via IP ranges published in their docs.

Key point: robots.txt is still the primary control surface for web crawling. It is not a legal contract. It is an access policy. You should pair it with rate limiting and verification.

Small business note: you do not need to memorize this list. Your hosting provider or CDN dashboard will show you which bots are hitting your site if you ever want to check. For most sites the practical decision is binary: allow AI crawlers (recommended for local businesses that want discovery) or block them (for publishers and data-heavy sites).

Part 3. Robots.txt patterns you can paste and adapt

Start from the simplest posture that matches your policy.

Allow everything (recommended for most local businesses)

If your goal is discovery and you have no proprietary content to protect, this is the right posture. You want AI assistants to know you exist.

# Allow all crawlers, including AI
User-agent: *
Disallow:

Sitemap: https://yoursite.com/sitemap_index.xml

That is it. If this describes your situation, you are done with this section, skip to Part 6 where the actual opportunity is.

Allow search, block training crawlers

# Allow normal search engines
User-agent: *
Disallow:

# Opt out of AI training bots
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Mixed policy by path

# Allow everything on marketing pages
User-agent: *
Allow: /blog/
Allow: /docs/

# Block training on members-only research
User-agent: GPTBot
Disallow: /research/
User-agent: Google-Extended
Disallow: /research/

Strict allowlist for unknown agents

If you are frequently scraped, flip the default.

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Add others you explicitly trust

Tips that prevent subtle errors:

• Host only indexable 200 pages in your XML sitemaps. Training bots often seed from sitemaps.
• Keep canonical links, hreflang, and internal links consistent. Conflicting signals cause duplication in both search and AI answer selection.
• Robots change fast. Calendar a quarterly review of vendor user-agents and IP verification steps.

Part 4. Beyond robots: your enforcement layer

This section assumes a CDN and access to server-level configuration. If you are on shared WordPress hosting without a CDN, skip to Part 5.

Robots rules work best when the crawler is cooperative. You still need verification and rate control.

• Reverse DNS verification
  Resolve the crawler’s IP address back to the domain that claims it. Many vendors publish instructions. Allow or deny based on verified reverse DNS.
• Rate limiting and burst control
  At the CDN or load balancer, set reasonable caps for new user agents and unknown IPs. If you run APIs or large file stores, add per token or per IP quotas.
• Header based policies
  Require a specific header or token for higher volume endpoints like site search APIs or feeds. Do not expose high value endpoints to anonymous traffic.
• Honey pages and traps
  Place unlinked canary URLs that only a crawler will discover. If an unapproved agent hits them, you have high confidence it is ignoring robots. Block and log.
• Abuse reporting
  Maintain a lightweight process to send abuse reports to vendors that ignore policy. Include IPs, timestamps, and requested URLs.

Part 5. Ethical, legal, and brand posture

Have a public page that states your AI use policy. Clarify what you allow for training and what you expect for citation and attribution. Include a contact email for research and licensing. This page helps your legal team, makes vendor conversations easier, and signals seriousness.

For your media assets, adopt Content Credentials (C2PA) where feasible. It does not prevent copying, but it embeds provenance and usage intentions in assets.

If you license data, watermark samples and use delayed or reduced feeds for unlicensed endpoints. Provide high quality feeds to paying partners only.

For small businesses: a public AI policy page is optional at your scale, but it costs an hour and it is a differentiator when clients ask about your practices. A short page stating what you allow and how you expect attribution is enough. You do not need legal review for a two-paragraph statement.

Part 6. Design your pages to be cited

AI systems prefer content that is easy to parse, specific, and defensible.

• Lead with the answer
  Open with a 2 to 4 sentence definitive answer and then go into details. Add a supporting table or bullet list. This format is consistently chosen by both search and AI assistants.
• Be the original
  Use your own data, images, and examples. When the model looks for authoritative facts, original numbers and named sources make your page safer to cite.
• Add structured data
  Use Article, Product, Organization, and Breadcrumb schema at a minimum. Add HowTo only where the content genuinely matches. Skip FAQPage schema, Google deprecated FAQ rich results in May 2026, so the markup no longer produces SERP features. The FAQ content is still highly valuable for AI citation; you just do not need the JSON-LD wrapper anymore. Ensure your Organization entity is consistent across your site with sameAs links to your official profiles.
• Stable anchors and headings
  Add IDs to section headings so a deep link can reference a specific claim. Example: <h2 id="pricing-method">How we calculate pricing</h2>. Some assistants surface anchor links directly.
• Author and review metadata
  Show a named author with a real bio and credentials. For health, finance, and legal content, show a reviewer with credentials and last review date.
• Visuals with descriptive alt
  AI agents often extract image context. Use descriptive alt attributes and filenames. Compress but retain legibility. Prefer SVG for diagrams where you can.
• Terms of use visible
  Link to clear terms that explain licensing and citation expectations.

The bilingual citation opportunity

Here is something almost nobody is doing yet: AI assistants answer questions in whatever language the user asked. Someone asking ChatGPT “¿cuánto cuesta un sitio web en Utah?” gets an answer synthesized from Spanish-language sources, and there are very few of those for most local markets. The competition for Spanish-language AI citation is close to zero in most US local markets right now.

If you publish in both languages, you are eligible for citation in both. The mechanics are the same as English: answer the question directly in the first two sentences, use clear headings, mark up your schema with the right inLanguage value, and connect the versions with hreflang so the engines understand they are pairs rather than duplicates.

In markets like the Salt Lake Valley, where a meaningful share of residents search in Spanish, this is the same arbitrage that exists in traditional SEO, just earlier and with less competition. Our local SEO strategies guide covers the traditional-search side of the same play.

Part 7. Build AI friendly content libraries

Create pages that act like reference docs for your domain. These libraries tend to be quoted by assistants.

• Lexicon pages
  Define every term in your niche on a single URL, with jump links and a short, precise definition for each entry. Our marketing and tech glossary is built this way: 130+ terms, each with a stable anchor so any single definition can be linked or cited directly.
• Process and decision trees
  Convert long paragraphs into step-by-step checklists and decision trees. Assistants love to lift them. Our Google Business Profile playbook uses this format throughout.
• Data pages
  Publish evergreen metrics and refresh quarterly. Keep the URL stable and show a revision history.
• FAQs with evidence
  Instead of fifteen thin FAQs, write seven that include a short answer, a table, and a reference to your deeper resource.
• Canonical comparisons
  Write side by side comparisons that are fair and specific. Assistants frequently cite balanced comparison pages.

Part 8. KPIs and measurement in the AI era

If you run a small business: your entire measurement program is the quarterly check described at the top of this article. Pick 10 to 20 questions your customers actually ask, run them through ChatGPT, Perplexity, and Google’s AI Overview once a quarter, and note whether you appear. Track one number: how many of those questions cite you. That is enough signal at your scale. The rest of this section is for teams with dedicated analytics resources.

1. Visibility
   • Inclusion in AI answer boxes where visible
   • Mentions or citations in assistants that show sources
   • Branded search demand and direct traffic trend as proxies

2. Compliance
   • Percentage of crawler hits that respect robots and rate limits
   • Number of incidents where an unapproved agent fetched restricted paths
   • Time to remediation when something breaks

3. Value
   • Assisted conversions from pages built for citation
   • Qualitative lift in sales conversations referencing your materials
   • Licensing revenue if you commercialize access to structured data or archives

To track visibility, create a quarterly manual sampling program:

• Define the 50 to 100 most important questions about your product and niche
• Check major assistants and search features with those questions
• Record when your brand or page is cited, the format used, and any anomalies
• Share a single KPI: percent of tracked questions where you are cited

Part 9. Server log checklist to monitor AI crawlers

Logs are your source of truth. Create a simple dashboard that updates daily. Enterprise / dev team section. Skip if you do not have log access.

• Top user agents by hits and bandwidth
• Requests per minute for new or unknown agents
• Hits to disallowed paths from robots aware agents
• Reverse DNS pass or fail rate
• Average response time during crawl spikes
• Honey page hits

Store 90 days of logs hot and six to twelve months cold. If you use Cloudflare, Fastly, or a similar CDN, stream logs to a warehouse. Ship Slack alerts when thresholds are crossed.

Part 10. RAG and API strategy

Many assistants use retrieval augmented generation. You can make your content easier to retrieve with fidelity. Relevant if you publish structured datasets or run a knowledge base. Skip otherwise.

• Provide a clean, documented JSON or CSV export for key datasets behind an authenticated key. Use ETag or Last Modified to minimize crawl.
• Offer a developer page that explains what is available and the license. Consider rate limited free keys for research and a paid tier for heavier use.
• If you run a knowledge base, offer an embeddable widget or sitemap of Q and A pages with lastmod dates.
• For partners, supply an embeddings ready corpus: one JSON object per passage with stable IDs, title, URL, and a short abstract.

Part 11. Governance for enterprises

Enterprise governance section. If you are a team of one to ten, the practical version is: decide your policy once, write it down, revisit it annually.

• Assign a single owner for AI crawler policy. Marketing should not change robots.txt without platform approval, and vice versa.
• Keep a changelog for robots, sitemaps, and crawl rules in your repo.
• Review partnerships quarterly. If you license data, align product, legal, and finance on pricing and terms.
• Train your support and PR teams on your AI policy so they can respond correctly to inquiries.

Part 12. Risk scenarios and how to respond

• Training opt out ignored
  Document IPs, timestamps, user agents, and requested paths. Block offending IPs at the edge. Publish an incident note internally and, if needed, escalate through vendor channels.
• Sudden traffic spike from a new bot
  Throttle the agent. If it respects 429 and robots, negotiate access rules. If not, treat as abusive.
• AI summaries misstate your claims
  Update the page with a clearer short answer and a date. Add a precise table or numbered list at the top. Publish a follow up explainer. Where possible, use feedback links in the product to report issues, then log a QA check 30 days later.
• Your site gets cited without a link
  Add visible source friendly components like tables and named statistics that are harder to paraphrase without attribution. Consider public requests for citation in your policy page.

Part 13. Checklists you can paste into tickets

Small business checklist (the whole thing)

• Decide: allow or block AI training crawlers. Write it into robots.txt.
• Add a short answer (2 to 4 sentences) at the top of every page that answers a question.
• Add IDs to your H2 headings so specific sections can be linked.
• Confirm Organization and Article schema are clean (test one page in Google’s Rich Results Test).
• Add a named author with a real bio to your articles.
• Once a quarter: ask AI assistants 10 to 20 customer questions, note whether you are cited.

That is the complete program at small-business scale. The checklists below are for larger teams.

Robots and enforcement

• Review vendor docs and update user agent controls
• Verify reverse DNS for high volume crawlers
• Add honey pages and alerts
• Document policy on a public page and link it in robots.txt as a comment

Site structure for citation

• Short answer at the top of every reference page
• Stable anchors on H2s
• Article or Product schema plus Organization with sameAs
• Author and reviewer metadata with last updated date
• Replace images without alt, add descriptive filenames

Monitoring and reporting

• Daily crawler dashboard in your warehouse or observability stack
• Quarterly AI citation audit on your 100 query set
• A single KPI per quarter for leadership, tied to revenue or risk

Part 14. Action plan for the next 30 days

The small business version (one afternoon)

• Set your robots.txt policy (15 minutes)
• Rewrite the opening two sentences of your five most important pages so they answer the page’s core question directly (2 hours)
• Run one page through Google’s Rich Results Test and fix whatever is broken (30 minutes)
• Run your first AI visibility check: 10 questions, three assistants, note the results (30 minutes)

Week 1

• Publish or update your AI policy page
• Add robots.txt entries for the training crawlers you intend to block or allow

Week 2

• Implement reverse DNS verification and edge rate limits
• Add honey pages and alerts

Week 3

• Convert five core pages into citation ready formats with short answers, tables, and anchors
• Add or clean Organization and Article schema

Week 4

• Build the first version of your crawler dashboard
• Run the first 100 query AI visibility audit and set a baseline

The bottom line

AI indexing is not a threat to manage so much as a channel to earn. The businesses getting cited by AI assistants are not the ones with the most sophisticated crawler policies; they are the ones whose pages answer questions clearly, cite real numbers, and are structured so a machine can extract the answer without guessing. That is the same work that has always made content good. The difference now is that the payoff shows up in two places instead of one.

If you want help getting your site structured for both search and AI citation, that is part of what we do for small businesses across the Salt Lake Valley, in English and Spanish. Get in touch if you want to talk it through.

Sources

• Google – AI features & your website: https://developers.google.com/search/docs/appearance/ai-features
• Google – Google Extended control: https://blog.google/technology/ai/an-update-on-web-publisher-controls/
• OpenAI – GPTBot: https://platform.openai.com/docs/bots
• Perplexity – Crawlers: https://docs.perplexity.ai/guides/bots
• Common Crawl – Opt‑out & robots: https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-