rule Technical SEO

Robots.txt Tester

Test whether a crawler can fetch a URL under your robots.txt rules.

Paste your robots.txt, pick a crawler, and test any URL or path. See whether it is allowed or blocked and exactly which rule decided it.

build_circle Try the free tool
list On this page expand_more
star

Key takeaways

  • check_circle Test your robots.txt before deploying, because a single misplaced slash can block an entire site from search.
  • check_circle The longest matching pattern decides the outcome, not the order of the lines in the file.
  • check_circle A named user-agent group replaces the wildcard group for that crawler instead of adding to it.
  • check_circle Disallow controls crawling only. It does not remove a URL from search results.
  • check_circle Patterns are case sensitive, so test the exact URL casing your site serves.
  • check_circle Test important URLs as Googlebot, Bingbot, and each AI crawler you have a policy for.
  • check_circle Keep rendering assets such as CSS and JavaScript crawlable so search engines see pages as visitors do.
rule

What is Robots.txt Tester?

A robots.txt tester answers one question: given this file, can this crawler fetch this URL? You paste the file contents, choose a crawler, and enter one or more URLs or paths. The tool returns Allowed or Blocked for each, along with the specific rule that decided it and its line number.

Robots.txt is a plain text file at the root of a domain. It follows the Robots Exclusion Protocol, now standardized as RFC 9309. Each group begins with User-agent lines, followed by Allow and Disallow rules that match URL paths. A crawler picks the group that names it most specifically, then applies the longest matching rule in that group. Because these rules interact, reading the file by eye is unreliable, and small edits can have large effects.

This tester is built for the moment before a change ships, and for debugging afterward. It supports the * and $ wildcards, treats Allow as the winner on equal-length ties, and includes common search and AI crawlers in a dropdown: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot, and the wildcard. Everything runs in your browser, so a draft file never leaves your device. It complements the Robots.txt Checker, which reviews a live file, and the Robots.txt Generator, which helps you write one.

insights

Why robots.txt testing matters

Robots.txt mistakes fail silently. A wrong rule does not throw an error, it just stops crawlers from reaching pages, and the damage shows up weeks later as lost impressions. Testing turns the file from guesswork into a checkable spec, so the people who edit it can prove key pages stay reachable.

arrow_forward A staging block copied to production can hide the whole site from crawlers.
arrow_forward AI crawler policies depend on group precedence that is easy to misread.
arrow_forward Overlapping Allow and Disallow rules produce surprising results without a test.
arrow_forward Blocked assets can stop Google from rendering pages correctly.
play_circle

How to use this tool

  1. 1

    Paste robots.txt

    Use your live file or a draft you are about to deploy.

  2. 2

    Choose a crawler

    Test as Googlebot, Bingbot, an AI bot, or the wildcard.

  3. 3

    Test URLs

    Enter paths or full URLs and read the verdict and the matching line.

fact_check

What this tool checks

verified

User-agent group selection

Finds which group applies to the crawler you pick. A named group such as GPTBot beats the wildcard group, and the wildcard only applies when no named group matches.

verified

Longest-match precedence

Compares every Allow and Disallow pattern in the chosen group against the path. The pattern with the most characters wins, which is how Google resolves overlapping rules.

verified

Allow versus Disallow ties

When an Allow and a Disallow pattern are exactly the same length, the tool lets Allow win. This matches the least restrictive rule behavior Google documents.

verified

Wildcard and end anchor

Supports * to match any run of characters and $ to anchor the end of a URL. Path matching is case sensitive, so /Shop/ and /shop/ are treated as different paths.

verified

Deciding line number

Reports the exact line of the rule that decided each verdict, with its text. If nothing matches, it says so and treats the URL as allowed, as crawlers do.

verified

Batch URL testing

Accepts full URLs or paths, one per line, up to 200 per run. Full URLs are reduced to path plus query string, so the host name is ignored.

engineering

Technical guide

Signals, standards, and what to fix when checks fail.

code

How a crawler picks its group

A robots.txt file is split into groups, each starting with one or more User-agent lines. A crawler obeys only the single group whose name matches it most specifically. If a GPTBot group exists, GPTBot ignores the wildcard group entirely, even for rules the wildcard group states. That surprises many teams: adding a short GPTBot group can accidentally lift, for GPTBot only, restrictions you set for everyone else. The tester mirrors this by choosing the longest matching agent name first and falling back to the asterisk group only when nothing else matches. Test each crawler separately, because the answer for Googlebot can differ from the answer for the wildcard.
code

Pattern matching in detail

Every rule path is matched from the start of the URL path, including the query string. Disallow: /private blocks /private, /private/, and /private-notes.html because matching is prefix based. A * matches any sequence of characters, and a trailing $ means the URL must end exactly there, so Disallow: /*.pdf$ blocks PDF files but not /file.pdf?download=1. An empty Disallow line matches nothing and allows everything. Paths are case sensitive, so check the exact casing your CMS produces. The tester escapes regular expression characters in your patterns, so a dot in a rule is treated as a literal dot.
code

Precedence between overlapping rules

When several rules match one URL, the most specific one decides, and specificity is measured by pattern length. Given Disallow: /private/ and Allow: /private/public/, the URL /private/public/page.html is allowed because the Allow pattern is longer. If the two patterns have equal length, Allow wins. This is why an Allow line can carve an exception out of a broad block, and why reordering lines in the file changes nothing. The tester shows the winning line number so you can confirm the exception you wrote is the one being applied.
code

What robots.txt does not control

Robots.txt governs crawling, not indexing. A blocked URL can still appear in search results as a bare link if other pages link to it, and Google cannot read a noindex tag on a page it is not allowed to fetch. Google stopped supporting a noindex directive inside robots.txt in 2019. The file also is not access control: it is public, and only well-behaved bots follow it. Google reads at most the first 500 KiB of the file. Directives such as Crawl-delay are ignored by Google, and this tester does not evaluate Sitemap or Crawl-delay lines at all.
school

Deep dive

groups

Getting user-agent groups right

Group selection is the most common source of wrong assumptions. A crawler reads only the group that best matches its name. If you write a wildcard group that blocks /private/ and then add a separate group for Googlebot that blocks only /tmp/, Googlebot no longer sees the /private/ rule at all. Rules do not merge across different groups. When you want a named crawler to follow the general rules plus something extra, repeat the general rules inside its group. Multiple User-agent lines stacked together share one rule set, which is a tidy way to give several crawlers the same policy. Always test the wildcard and every named crawler you care about, because each may resolve differently.

Stacked agent lines

Consecutive User-agent lines with no rule between them form one group. Rules that follow apply to every listed crawler, which keeps shared policies short and consistent.

Named group replaces wildcard

Adding a GPTBot group means GPTBot ignores the wildcard group. Copy any shared blocks into the named group if you still want them to apply. This is the most common cause of surprise crawl access.

Agent name variants

Search engines run several crawlers, such as Googlebot-Image and Googlebot-News. Without a group of their own they fall back to the Googlebot group, and then to the wildcard. Test the variant you care about as well.

code

Designing patterns that do what you mean

Most robots.txt patterns are prefix matches, so short patterns reach further than expected. Disallow: /shop blocks /shop, /shop/, /shopping-guide/, and /shop-locations. Add a trailing slash when you mean a folder. Use * sparingly for parameters, for example Disallow: /*?sessionid= to block session URLs while leaving the clean versions open. Use $ when file endings matter. Keep the file small and readable, with comments explaining why each block exists. Google reads only the first 500 KiB, so very long files risk being cut off. After every pattern change, rerun a fixed list of URLs that must stay allowed and URLs that must stay blocked, and treat any change in results as a review item.

Trailing slash discipline

A slash at the end of a folder rule prevents accidental matches on similarly named pages. Compare Disallow: /blog and Disallow: /blog/ to see how far each reaches.

Parameter blocking

Wildcards can block tracking or sort parameters. Confirm they do not also block canonical pages that legitimately carry a query string, such as paginated category URLs. Test both a blocked and a clean URL.

Regional folders

Sites with /uk/, /au/, or /sg/ folders should test each regional root and key page separately, since a broad rule written for one folder can catch another.

smart_toy

Setting policy for AI crawlers

Many AI companies publish crawler names and say they honor robots.txt. GPTBot and OAI-SearchBot belong to OpenAI, ClaudeBot to Anthropic, PerplexityBot to Perplexity, and Applebot to Apple. Google-Extended is a control token rather than a separate crawler: it tells Google whether content may be used for its generative AI products, while normal Googlebot crawling for Search continues. Decide by crawler and by purpose. You may want search-oriented bots allowed so your pages can be cited, and training-oriented bots restricted. Write each choice as an explicit group, then test key URLs against each name. Remember that robots.txt is a request to compliant bots, not enforcement, so pair it with server-level controls if you need a hard block. Revisit the policy whenever a vendor announces a new crawler.

Search versus training

Some vendors run separate crawlers for search features and for model training. Blocking one does not automatically block the other, so list each name you intend to control.

Google-Extended token

Disallowing Google-Extended limits AI use of your content but does not change how Googlebot crawls for Search. Test it as its own entry to confirm the group you wrote resolves.

Keep a policy record

Write the reason for each AI rule in a comment and store it in version control. Vendor crawler names change, so review the list a few times a year. Assign an owner for the policy.

verified

A safe robots.txt release workflow

Treat robots.txt like production code. Keep it in version control, require a second reviewer, and maintain a standing test list of URLs with expected results: allowed for homepage, service pages, product pages, and sitemaps, blocked for admin, internal search, and cart paths. Before deploy, paste the draft into the tester and confirm every expected result. After deploy, fetch the live file, confirm it matches the reviewed draft, and check the robots.txt report in Google Search Console. Watch for CDN and edge layers that serve a different file than your origin, and for staging settings that slip through during launches. A rollback copy of the previous file makes any mistake a two minute fix.

Standing test list

Keep a short spreadsheet of URLs and expected verdicts. Running it before each release makes regressions obvious and takes only minutes. Include at least ten URLs covering every major template and folder.

Post-deploy verification

Fetch the live file after deployment and compare it with the reviewed draft. Confirm the status code is 200, since an error response can change how crawlers treat your site.

Monitor after changes

Watch crawl stats and indexed page counts in Search Console for a few weeks after a significant robots.txt change, and investigate any sudden drop in crawled pages. Compare it against the pre-change baseline.

compare

Examples

thumb_up Strong examples

Block a folder, allow one page

Disallow: /private/ + Allow: /private/public/

The longer Allow pattern wins for URLs inside the public folder, so you can open one exception without rewriting the broader block. It is the simplest pattern for exceptions.

Block one AI crawler only

User-agent: GPTBot / Disallow: /

A named group applies only to that crawler. Googlebot and Bingbot keep following the wildcard group and are not affected. This is a common way to set an AI crawler policy.

Block PDFs without touching pages

Disallow: /*.pdf$

The end anchor limits the rule to URLs that finish with .pdf, so HTML pages and other file types remain crawlable. Always test a sample URL with a query string too.

thumb_down Weak examples

Whole site blocked

User-agent: * / Disallow: /

This is the standard staging file. On a live site it stops every compliant crawler from fetching any page, and rankings can fall quickly. Always test before deploy.

Rendering assets blocked

Disallow: /wp-content/

Themes, scripts, and images often live here. Blocking the folder can prevent Google from rendering pages fully, which hurts how they are evaluated. Check rendering after any change.

Wrong path casing

Disallow: /Private/

Paths are case sensitive, so this does not block /private/. The folder stays crawlable while the file looks like it is protected.

rule

Best practices and common mistakes

check_circle Best practices

  • done Test your homepage, top service pages, product templates, and XML sitemap URL as Googlebot before every robots.txt release.
  • done Test the same URL as the wildcard and as each named AI crawler, because a named group replaces the wildcard group instead of adding to it.
  • done Add a test for one URL that should be blocked, such as /wp-admin/ or an internal search path, to confirm your blocks still work.
  • done Keep CSS, JavaScript, and image folders crawlable so Google can render pages the way visitors see them.
  • done Use the deciding line number from the tester to remove or narrow the rule, rather than adding a competing Allow line on top.
  • done Save the previous robots.txt in version control and re-run the same test list after every deploy or CDN change.

cancel Common mistakes

  • close Deploying a staging robots.txt with Disallow: / to production, which blocks the entire site from compliant crawlers.
  • close Using Disallow to remove a page from search results, since it only stops crawling and can hide a noindex tag from Google.
  • close Assuming a rule for the wildcard also covers GPTBot or ClaudeBot once a named group for that bot exists.
  • close Blocking /wp-content/, /assets/, or script folders that pages need for rendering and rich results.
  • close Relying on robots.txt to protect private files, since the file is public and only polite bots obey it.
lightbulb

Common use cases

task_alt

Verifying that a robots.txt draft still allows key money pages before it replaces the live file.

task_alt

Debugging why Search Console reports a URL as blocked by robots.txt and finding the exact line responsible.

task_alt

Checking that a new AI crawler policy blocks GPTBot but still allows Googlebot and Bingbot on the same paths.

task_alt

Reviewing an ecommerce faceted navigation block to confirm parameter URLs are blocked while category pages stay open.

task_alt

Confirming a migration or CDN change did not alter crawl access for regional folders such as /uk/ or /au/.

groups

Who should use this

person Technical SEOs person Web developers and DevOps engineers person Site owners preparing a launch or migration person Agencies auditing client sites person Content teams setting an AI crawler policy
dictionary

Glossary

User-agent
The line that names which crawler a group of rules applies to. The value * means any crawler that has no more specific group of its own.
Disallow
A rule telling a crawler not to request URLs whose path begins with the given pattern. An empty Disallow value blocks nothing.
Allow
A rule that permits crawling of a path, usually used to create an exception inside a broader Disallow block.
Longest match
The precedence rule where the matching pattern with the most characters wins when several Allow and Disallow rules apply to the same URL.
Wildcard
The asterisk character in a rule path, matching any sequence of characters, useful for parameters and file extensions.
End anchor
The dollar sign at the end of a pattern, requiring the URL to end at that exact point, such as .pdf$.
Crawl budget
The amount of crawling a search engine is willing and able to spend on a site, which robots.txt rules can help direct.
RFC 9309
The formal specification of the Robots Exclusion Protocol, which defines group selection, matching, and how crawlers should treat errors when fetching the file.
quiz

Frequently asked questions

Ready for a deeper audit?

HeyLead reviews your site, competitors, and growth opportunities with no sales pressure.

Get a Free Audit arrow_forward