admin_panel_settings AI Visibility & GEO

AI Bot Access Builder

Choose which AI crawlers can use your content and get the robots.txt rules.

Decide bot by bot which AI crawlers may train on your content, index it for AI search, or fetch it for users. Get copy-ready robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more.

build_circle Try the free tool
list On this page expand_more
star

Key takeaways

  • check_circle AI crawlers fall into three groups, training, search, and user fetch, and each group deserves its own decision.
  • check_circle Blocking training bots does not remove you from AI search, because search crawlers use separate user agents.
  • check_circle Google-Extended and Applebot-Extended are control tokens and do not change ordinary search crawling.
  • check_circle Robots.txt is a voluntary request, so private content needs authentication and not just a Disallow rule.
  • check_circle A bot with its own named group ignores the wildcard group, so protected paths must be repeated in that group.
  • check_circle Merge generated rules into your existing file and verify the live robots.txt after every change.
  • check_circle Review the policy each quarter, since AI vendors add and rename crawlers often.
admin_panel_settings

What is AI Bot Access Builder?

AI Bot Access Builder is a free tool that turns a set of decisions about AI crawlers into a robots.txt block you can copy or download. You choose a starting policy, then set each of 15 named crawlers to Allow, Block, or Skip. The tool labels every bot as training, search, or user fetch so the decision matches the bot's real job.

The distinction matters because AI companies operate several crawlers with different purposes. A training crawler gathers text to build models. A search crawler indexes pages so an AI product can cite them. A user-fetch agent retrieves a page live when a person asks a question about it. Blocking all of them removes you from AI answers, while allowing all of them gives up control over training use. The builder lets you split the two.

The output follows standard robots.txt grouping. Allowed bots share one group, blocked bots share another, and an optional wildcard group covers everything else. You can add protected paths, such as checkout or account pages, and a sitemap line. The tool also shows a live count of allowed, blocked, and skipped bots as you work. The page runs entirely in your browser, so nothing you enter is sent anywhere. The result is a starting point to merge into your live file, not a replacement for it.

insights

Why AI crawler access matters

More people now ask an AI assistant a question before they open a search results page, and those assistants build answers from crawled content. Your robots.txt is the first place a crawler learns what it may use, so a careless rule can quietly remove your brand from answers or give away content you meant to license.

arrow_forward A single misplaced Disallow can remove a whole site from AI search citations.
arrow_forward Training and search use different crawlers, so one blanket rule rarely fits both goals.
arrow_forward Publishers and content businesses need a documented position on training access.
arrow_forward Crawler names change often, so a policy written once goes stale within months.
play_circle

How to use this tool

  1. 1

    Pick a policy

    Start from the default (allow AI search and user fetches, block training), allow all, or block all. Then change any single bot.

  2. 2

    Set bot by bot

    Each crawler is labeled training, search, or user fetch, so you can allow citations while blocking training.

  3. 3

    Copy the rules

    Merge the output into your robots.txt, deploy, and confirm with the Robots.txt Tester.

fact_check

What this tool checks

verified

Bot catalog by purpose

Lists 15 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, DuckDuckGo, ByteDance, and Common Crawl, each labeled training, search, or user fetch so every choice reflects what the bot actually does.

verified

Policy presets

Starts from allow all, allow search and user fetches while blocking training (the default), block all, or custom. Changing any single bot switches the preset to custom automatically so your edits are never overwritten.

verified

Allow, Block, or Skip per bot

Each crawler gets one of three states. Skip leaves the bot out of the output entirely, so it falls under your existing rules or the wildcard group instead of getting its own group.

verified

Protected paths

Paths you list, one per line, are written as Disallow lines in the allowed bot group and in the wildcard group, so areas like /checkout/ or /private/ stay closed to every crawler.

verified

Wildcard default group

An optional User-agent: * group, on by default. It allows everything else, or applies your protected paths, so crawlers you did not list keep behaving the way they do today.

verified

Sitemap line and export

Adds an optional Sitemap line, shows a live count of allowed, blocked, and skipped bots, and lets you copy or download the result as robots.txt. Everything runs in your browser.

engineering

Technical guide

Signals, standards, and what to fix when checks fail.

code

How the generated file is structured

The tool writes one group for all allowed bots, one group for all blocked bots, and an optional wildcard group. Robots.txt allows several User-agent lines to share a single rule set, so the file stays short. The allowed group ends with Allow: / after any protected-path Disallow lines. The blocked group is a plain Disallow: /. Because Google and most other parsers apply the most specific matching rule, protected paths still win over the broad Allow. Rules for a named bot replace, rather than add to, the wildcard group for that bot, which is why protected paths are repeated in each allowed group.
code

Training, search, and user-fetch tokens

Training crawlers such as GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider, and CCBot collect content to build or improve models. Search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot index pages so an AI product can cite them. User-fetch agents such as ChatGPT-User, Claude-User, and Perplexity-User retrieve a page when a person asks about it. Google-Extended and Applebot-Extended are control tokens, not separate crawlers: they tell Google and Apple whether already-crawled content may be used for AI features, and they do not change normal search crawling. Treat each token as its own decision and record the operator's documentation link beside it.
code

Merging with an existing robots.txt

Do not paste the output over a live file. Keep your existing groups for Googlebot, Bingbot, and any CMS rules, then add the AI groups below them. Remember that a crawler follows only the most specific group that names it, so a User-agent line for GPTBot means GPTBot ignores your wildcard group entirely. Keep one Sitemap line per sitemap and place them anywhere in the file. After deploying, fetch yourdomain.com/robots.txt in a browser, confirm it returns 200 as plain text, and keep the file under the 500 KiB limit Google enforces.
code

What robots.txt cannot do

Robots.txt is a voluntary standard. Major operators state that their named crawlers honor it, but the file does not authenticate anyone, and it does not remove content that was collected earlier. Some operators say user-initiated fetchers may behave differently from background crawlers, so check each vendor's current documentation. Anything confidential belongs behind a login, and firewall or CDN rules can enforce a block that a poorly behaved bot ignores. Verify enforcement in your server logs, for example with the AI Crawler Log Analyzer, rather than assuming the file worked. Treat robots.txt as one layer of protection, not the whole plan.
school

Deep dive

account_tree

Training, search, and live fetch

Most confusion about AI crawlers comes from treating them as one thing. Operators publish separate user agents for separate jobs, and the difference decides whether a block protects you or hurts you. A training crawler collects text that may shape future models. A search crawler builds an index that an AI product consults when it composes an answer with links. A user-fetch agent runs only when a person pastes a URL or asks a question that needs a live page. If your goal is to be recommended, search and user fetch are the ones that bring visibility. If your goal is to control how your writing is reused, training is the lever. The builder shows the label beside every bot so you can make each call separately, and the search-only preset encodes the most common split.

Where visibility comes from

Citations in AI answers depend on search and user-fetch access. Blocking those bots removes you from the source list, even if you welcome training use. Keep those bots allowed on every page you want cited.

Where control comes from

Blocking training crawlers signals that content should not be used to build models. It does not undo earlier collection, and enforcement depends on operator compliance. Pair the block with a licensing position if the content is commercially valuable.

Mixed policies are normal

Many sites allow search bots, block a few training bots, and skip the rest. That is a legitimate policy, and it can be documented and reviewed each quarter. The builder's Skip option helps when you want to defer a decision.

rule

Robots.txt matching rules

Robots.txt looks simple, but a few matching rules cause most mistakes. A crawler picks the single group whose User-agent line best matches its name and ignores all other groups, including the wildcard. Inside that group, the longest matching path wins, and Allow beats Disallow when the lengths tie. Paths are case sensitive and match from the start of the URL path. Comments start with a hash and are ignored. The file must live at the root of each host, so www and non-www, and each subdomain, need their own copy. Google reads the first 500 KiB and ignores the rest, so very long files can silently lose rules at the end. The builder's output respects these rules, but your existing file may not, so read the merged result once from top to bottom before publishing.

One group per crawler

If GPTBot has its own group, it never reads User-agent: *. Repeat any shared Disallow lines inside each named group that needs them. Forgetting this is the most common AI robots.txt bug.

Longest match wins

A Disallow: /checkout/ inside a group that also says Allow: / still blocks the checkout, because the longer path is more specific and takes priority. Test unusual paths with a robots.txt tester before deploying.

One file per host

Each hostname serves its own robots.txt. A regional subdomain or a separate country domain needs its own copy of the AI rules. A missing file is treated as allow everything.

verified

Enforcement and verification

A robots.txt rule is only as good as the crawler that reads it. Established operators document their user agents and state that they honor the file, and many publish IP ranges so you can verify a visitor. Less scrupulous scrapers ignore both, and some send a browser user agent. For a hard block you need a firewall or CDN rule, ideally based on published IP ranges rather than user agent strings alone. Verification closes the loop. After deploying, sample your access logs a week later and confirm blocked bots stopped requesting pages and allowed bots receive 200 responses. Cloud firewalls and bot-protection features can block AI search bots by accident, so a permissive robots.txt does not guarantee access. Logs show what actually happened.

Verify with logs

Paste a week of logs into the AI Crawler Log Analyzer to see hits, status codes, and top URLs for each named bot before and after your change. Repeat the check after every robots.txt change.

Firewall conflicts

A CDN or security rule that challenges unknown bots can return 403 to a search crawler you have allowed. Check status codes, not only robots.txt. Ask your host whether bot protection covers AI user agents.

IP verification

User agent strings can be spoofed. Operators publish IP ranges or reverse DNS patterns, and those are the reliable way to confirm a real visit. Use them for security decisions and treat user agents as labels only.

language

Regional sites and policy governance

Sites that serve the US, UK, Australia, and Singapore often sit on one domain with path prefixes, or on separate country domains. Path prefixes share one robots.txt, so one AI policy covers every region. Separate hostnames each need their own file, and it is easy for a regional file to drift from the main one. Rules about text and data mining, copyright, and licensing differ by country, so the decision to allow training is a business and legal choice, not a technical default. Record who approved the policy, the date, and the reasons. Add the review to a quarterly SEO checklist, because vendors introduce new crawlers and retire old ones. A short policy document plus a generated robots.txt is enough governance for most teams.

One domain, many regions

With /uk/, /au/, and /sg/ folders, a single robots.txt governs everything. Keep protected paths generic enough to cover each regional folder. Regional differences belong in content, not crawler rules.

Separate country domains

Each hostname needs its own robots.txt. Generate once, then copy the same AI block to every domain and confirm each one loads. Drift between copies is a common cause of surprises.

Document the decision

Store the policy, approver, and review date beside your SEO documentation so future changes start from a known baseline. Include which bots are allowed, blocked, and skipped. This keeps the policy easy to audit later.

compare

Examples

thumb_up Strong examples

Block training, allow search

User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /

GPTBot is blocked for training while OAI-SearchBot stays allowed, so pages remain eligible for ChatGPT search citations. Each bot has its own named group, so the intent is unambiguous.

Protected paths in an allowed group

User-agent: PerplexityBot Disallow: /checkout/ Allow: /

The bot may read public pages but not the checkout. Because the named group overrides the wildcard, the private path is repeated here instead of relying on User-agent: *.

Sitemap line kept

Sitemap: https://example.com/sitemap.xml

A Sitemap line is independent of user agent groups and helps every compliant crawler find your URLs. Keeping it while adding AI groups avoids accidentally dropping discovery signals.

thumb_down Weak examples

Blanket block for everyone

User-agent: * Disallow: /

This blocks Googlebot and every AI search bot along with training crawlers. It is a common leftover from staging and removes the site from both search and AI answers.

Invented or misspelled token

User-agent: OpenAI-Bot Disallow: /

Crawlers only match the exact user agent tokens they publish. An invented name matches nothing, so the rule silently does nothing while you believe the site is protected.

Named group without protected paths

User-agent: ClaudeBot Allow: /

ClaudeBot now ignores the wildcard group, so any Disallow rules you had there no longer apply to it. Private paths become open to that crawler unless they are repeated.

rule

Best practices and common mistakes

check_circle Best practices

  • done Start from the default policy. It allows AI search and user fetches and blocks training, so you stay eligible for AI citations. Adjust from there.
  • done Write down the reason for each blocked bot, such as licensing or bandwidth, so the next person to edit the file understands the intent.
  • done Merge the output into your existing robots.txt instead of replacing it, and keep every Sitemap line already in place.
  • done List private paths like /checkout/, /account/, and /staging/ in the protected paths box so they stay closed for every bot.
  • done Check the file with a robots.txt tester after deployment and confirm the live URL returns 200 as text/plain.
  • done Review server logs monthly for user agents that appear in traffic but are missing from your file, then decide on each one deliberately.

cancel Common mistakes

  • close Choosing block all as a safe default and then wondering why your brand never appears in AI answers.
  • close Blocking Google-Extended in the belief that it removes you from Google Search, which it does not.
  • close Relying on a robots.txt block to protect paid, private, or personal content that needs authentication.
  • close Leaving a staging robots.txt with Disallow: / on the live site after a launch.
  • close Adding a bot group and forgetting that the named group overrides the wildcard rules for that crawler.
lightbulb

Common use cases

task_alt

Deciding which AI vendors may train on a publisher archive while keeping articles eligible for citations in AI search.

task_alt

Setting a consistent AI crawler policy across US, UK, Australian, and Singaporean sites before a redesign.

task_alt

Preparing a client-facing AI access policy that lists each bot, its purpose, and the decision taken.

task_alt

Opening a new store or SaaS site to AI search bots so product pages can be cited from launch.

task_alt

Tightening access after a log review shows heavy crawling by a bot that brings no referral value.

groups

Who should use this

person Marketing leads deciding how the brand appears in ChatGPT, Claude, Perplexity, and Gemini. person SEO managers who own robots.txt and need a safe way to add AI crawler rules. person Publishers and media teams weighing training access against licensing plans. person Agency account managers documenting an AI crawler policy for several clients. person Developers who want copy-ready rules without hand-writing user agent tokens.
dictionary

Glossary

User agent
The name a crawler sends to identify itself. Robots.txt groups match on this token, such as GPTBot or ClaudeBot.
Training crawler
A bot that collects web content to build or improve AI models, for example GPTBot, ClaudeBot, or CCBot.
Search crawler
A bot that indexes pages so an AI product can retrieve and cite them, such as OAI-SearchBot or PerplexityBot.
User-fetch agent
A bot that retrieves a specific page when a person asks an AI assistant about it, such as ChatGPT-User or Claude-User.
Control token
A robots.txt name like Google-Extended that sets permission for AI use without being a separate crawler that visits your site.
Wildcard group
The User-agent: * rule set that applies to crawlers without a more specific group of their own.
Disallow
A robots.txt directive that asks matching crawlers not to fetch a path. It is a request, not an access control.
Common Crawl
A nonprofit open web archive, collected by CCBot, that many model builders use as a source of training text.
quiz

Frequently asked questions

Ready for a deeper audit?

HeyLead reviews your site, competitors, and growth opportunities with no sales pressure.

Get a Free Audit arrow_forward