terminal AI Visibility & GEO

AI Crawler Log Analyzer

Paste server logs and see which AI crawlers visit, what they fetch, and what fails.

Find out whether GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers actually reach your site. See hits per bot, top URLs, and error responses, all analyzed privately in your browser.

build_circle Try the free tool
list On this page expand_more
star

Key takeaways

  • check_circle Server logs are the only reliable record of which AI crawlers reached a page and what response they received.
  • check_circle A crawler that receives 403, 429, or 5xx responses cannot cite your content, whatever robots.txt allows.
  • check_circle Compare log samples before and after a robots.txt or firewall change to confirm the change worked.
  • check_circle User agent strings can be spoofed, so verify IP ranges before making security decisions.
  • check_circle A short log window can hide crawlers that visit weekly, so use at least seven days where possible.
  • check_circle Top URLs show what AI systems actually read, which often differs from what you want them to read.
  • check_circle Analyzing logs in the browser keeps visitor IP data on your machine.
terminal

What is AI Crawler Log Analyzer?

AI Crawler Log Analyzer is a free tool that reads raw web server access logs and reports which AI crawlers visited your site. You paste log lines, and the tool matches 23 known user agent tokens, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, and PerplexityBot. For each crawler it shows total hits, the number of days seen, response status counts, and the five most requested URLs.

Logs matter because AI crawlers do not appear in the analytics tools most marketers use. Those tools depend on JavaScript running in a browser, and bots rarely execute it. Your server, CDN, or host still records every request, including the user agent, the path, and the status code returned. That makes logs the ground truth for whether a crawler reached a page, how often, and whether it received content or an error.

The analysis runs entirely in your browser. The tool reads up to 200,000 lines, and nothing is uploaded or stored. It is designed for quick diagnosis: is GPTBot reaching the pricing page, is PerplexityBot hitting 404s, did a blocked crawler stop. It reads the user agent claimed in each request and does not verify source IP addresses, so treat it as a visibility check, not a security tool. Use it alongside robots.txt tools and the AI Bot Access Builder.

insights

Why AI crawler logs matter

Being cited by an AI assistant starts with being fetched. If a crawler never reaches your pages, or reaches them and receives errors, no amount of content work will earn a mention. Logs show that first step directly, letting you spot blocked bots, wasted crawling, and pages that are never read.

arrow_forward Firewalls and bot-protection rules can block AI search bots without any visible warning.
arrow_forward Error responses served only to crawlers never show up in normal user testing.
arrow_forward Crawl frequency hints at how much attention each AI system pays to your site.
arrow_forward Before-and-after log samples prove whether a robots.txt change worked.
play_circle

How to use this tool

  1. 1

    Export your access logs

    Pull a day or a week of raw logs from your host, server, or CDN.

  2. 2

    Paste and analyze

    We match known AI crawler user agents and count hits, status codes, and top URLs.

  3. 3

    Fix what fails

    Repair 4xx and 5xx responses on pages you want cited and confirm robots.txt is not blocking them.

fact_check

What this tool checks

verified

Known crawler matching

Scans each line, case-insensitively, for 23 user agent tokens, including GPTBot, ClaudeBot, PerplexityBot, and Googlebot. When several tokens match one line, the longest and most specific name wins.

verified

Hits per bot

Counts requests for every crawler found and ranks them from busiest to quietest. Summary tiles split AI crawler hits from search crawler hits. Each bot card shows operator, kind (training, search, or user fetch), status counts, and top URLs.

verified

Status code breakdown

For each bot, splits responses into 2xx, 3xx, 4xx, and 5xx counts, and flags any bot that received errors so you can see which crawlers are being served failures.

verified

Top requested URLs

Lists the five most requested paths for each bot with counts. Query strings are removed before counting, so parameter variants of the same page combine into a single line.

verified

Days observed

Counts the distinct calendar dates each bot appears on, read from the bracketed date in standard access logs. This shows whether a crawler visits steadily or appeared once.

verified

Private, in-browser parsing

Reads up to 200,000 pasted lines inside your browser tab. Nothing is uploaded, stored, or sent to HeyLead, which makes it safe for logs that contain visitor IP addresses.

engineering

Technical guide

Signals, standards, and what to fix when checks fail.

code

Supported log formats

The parser looks for the request section in quotes, such as "GET /pricing/ HTTP/1.1", followed by a three-digit status code. That matches Apache and Nginx combined log format and most CDN exports that keep the same layout. Dates are read from the square-bracket timestamp, like [12/Mar/2026:06:25:24 +0000]. Bot tokens are matched as whole tokens in the user-agent field. On combined-log lines that is the quoted value after the request and referrer, so a later x-forwarded-for field is not treated as the agent. If more than 200,000 lines are pasted, the extra lines are ignored and a notice is shown.
code

How user agents are matched

On combined-log lines the quoted user-agent field (after the request and referrer) is searched for the published token of every known crawler, as a whole token so a path like /gptbot-study/ does not count. Nginx logs that add x-forwarded-for after the agent still match. Tokens include OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User, Anthropic's ClaudeBot, Claude-SearchBot, and Claude-User, plus PerplexityBot, Amazonbot, Bytespider, CCBot, Diffbot, and others. The list also includes Googlebot, bingbot, and Applebot so you can compare AI crawl volume with traditional search bots. Sample a few flagged lines by hand when a number looks odd.
code

Reading status codes

A 200 means the crawler received the page. A 301 or 302 means it was redirected and will need a second request. A 404 or 410 means the URL is gone, and a 403 often signals a firewall or bot-protection rule. A 429 means rate limiting, and 5xx errors point to server trouble. AI bots cannot cite what they cannot fetch, so a crawler that mostly receives 403 or 429 is effectively blocked even if robots.txt says allow. Group errors by URL pattern and fix the templates or rules behind them, rather than chasing pages one at a time.
code

Spoofing and verification

Anyone can send a user agent string that claims to be GPTBot, so a log line proves only what the client said about itself. That is fine for measuring visibility, but not for security decisions. Major operators publish IP ranges or reverse DNS patterns you can check against the source addresses in your logs. Treat unusual volume from a claimed AI bot as a prompt to verify the IP, not as proof. If you decide to block traffic, use the verified ranges in a firewall rule. This tool does not perform IP checks.
school

Deep dive

query_stats

Why logs beat analytics for bots

Analytics platforms record a visit when a script runs in a browser. AI crawlers request HTML and usually stop there, so they leave no trace in dashboards and are hard to distinguish from noise. Your server logs record every request regardless of script execution, including the exact time, the path, the response code, and the user agent. That makes them the only place to answer basic questions: did this crawler come at all, what did it request, and what did we send back. Logs are also where you see problems that human testing misses, such as a security rule that returns 403 only to unknown bots. If you use a CDN, decide whether to analyze edge logs or origin logs. Edge logs show all traffic, while origin logs show only what missed the cache.

What analytics cannot show

Bot fetches, error responses, and blocked requests are invisible in JavaScript-based analytics, so a site can look healthy while crawlers fail. Only logs record them.

Edge versus origin logs

CDN logs include cached responses and blocked requests. Origin logs show only requests that reached your server, so counts differ between them. Pick one source and stay consistent. Note which one you used in reports.

Choosing a time window

Use at least seven days. Crawl patterns are uneven, and one day can capture a spike or miss a weekly visitor entirely. Longer windows also smooth out holiday dips. Say which window each report used.

table_chart

Reading the results

Start with the summary tiles. Lines scanned is your sample size. AI crawler hits and Search crawler hits show how much of it came from known crawlers. Then read each bot card. The hit count shows volume, and the days figure shows consistency. The status line matters most: a healthy crawler mostly gets 2xx, with a few 3xx for legitimate redirects. Errors are highlighted in red so you can check those URLs. Finally, review the top paths. They show what each system reads most, which is often your homepage, robots.txt, sitemap, and a handful of popular guides. If key commercial pages never appear, improve internal links and sitemap coverage, and check whether an upstream rule blocks them.

Volume and consistency

High hits over one day may be a single burst. Hits across many days indicate a regular crawl relationship. Compare against the previous month for context. A steady line matters more than a spike.

Error share

A few 404s are normal. A large share of 4xx or 5xx responses to one bot means something is broken or blocking it. Sort by bot to find the source.

Missing pages

When important pages are absent from top URLs, check internal links, sitemap inclusion, and whether robots.txt or a firewall restricts them. Fix the cause before requesting a recrawl. Re-run after fixing.

build

Fixing what fails

Most crawler errors trace back to a small set of causes. A 404 is usually a stale link or a removed page: redirect it with a 301 to the closest replacement, and update internal links. A 403 points to a firewall, WAF, or hosting rule, so allow verified crawler IP ranges or adjust bot-protection settings. A 429 means rate limits are too strict for a legitimate crawler, so raise limits or exempt verified bots. A 5xx suggests server capacity or code errors, and repeating patterns at certain times usually mean the origin cannot handle uncached requests. After each fix, collect a fresh log sample and compare the counts. Change one thing at a time, so you can tell which change worked.

Redirect and repair

Use 301 redirects for moved content and remove links to retired pages, so crawlers stop spending requests on dead ends. Recheck the same URLs a week later.

Loosen security rules carefully

Allow published crawler IP ranges for search bots you want. Do not simply disable protection for any client that says it is a bot. Document every exemption you add.

Cache to protect the origin

Well-cached pages return quickly to crawlers and avoid the 5xx errors that appear when many uncached pages are requested at once. Caching also lowers hosting costs.

trending_up

Reporting and regional sites

A monthly AI crawl summary is easy to build from this tool. Record the window, the number of bot requests, which crawlers appeared, the error share for each, and the pages they read most. Keep the same format each month so trends stand out. For regional sites serving the US, UK, Australia, and Singapore, filter logs by folder or hostname before pasting, so you can see whether crawlers reach /uk/, /au/, and /sg/ pages or only the US version. Crawlers often start with the main domain and reach regional pages later. If regional pages are missing, check the sitemap, hreflang links, and internal linking. Remember that the report shows access, not answers: it says a crawler read a page, not that an AI system cited it.

A simple monthly template

Record window, bot requests, top three crawlers, error share, and top pages. Consistency matters more than detail. Paste the summary numbers into a shared sheet.

Filtering by region

Filter lines by path prefix or hostname before pasting, then compare AI crawler coverage across your US, UK, Australia, and Singapore pages. A simple text filter on the export is enough. Run the tool once per region.

Access is not citation

A crawler visit shows that content was fetched. Use AI search audits and manual prompts to check whether pages are actually cited. Both checks together give the full picture.

compare

Examples

thumb_up Strong examples

AI bot receiving a clean page

203.0.113.5 - - [12/Mar/2026:06:25:24 +0000] "GET /pricing/ HTTP/1.1" 200 5316 "-" "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"

A named AI crawler fetched an important page and received a 200. This is the healthy pattern: the bot reached the content and the server delivered it without a redirect or error.

Crawler fetching a guide

203.0.113.6 - - [12/Mar/2026:06:25:31 +0000] "GET /blog/geo-guide/ HTTP/1.1" 200 8841 "-" "Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])"

A content page is being read with a full-size response. Repeated hits like this across several days suggest the page is in the crawler's regular rotation.

Steady daily presence

OAI-SearchBot: 214 hits, 2xx 208, 3xx 4, 4xx 2, 5xx 0, 7 day(s)

Hits spread across seven days with a tiny error share indicate a stable crawl relationship. Small 4xx counts usually reflect old URLs you can redirect or retire.

thumb_down Weak examples

Bot receiving a 404

203.0.113.7 - - [12/Mar/2026:07:02:10 +0000] "GET /old-page/ HTTP/1.1" 404 512 "-" "Mozilla/5.0 (compatible; PerplexityBot/1.0)"

The crawler followed a link or old index entry to a page that no longer exists. Redirect the URL to its replacement, and fix internal links that still point to it.

Firewall refusing a search bot

"GET /services/ HTTP/1.1" 403 199 "-" "Mozilla/5.0 (compatible; OAI-SearchBot/1.0)"

A 403 to a search crawler means a WAF or bot rule is blocking it. The page cannot be indexed for AI search however permissive your robots.txt is.

Rate limit on a citation bot

"GET /pricing/ HTTP/1.1" 429 0 "-" "Mozilla/5.0 (compatible; Perplexity-User/1.0)"

A 429 to a user-fetch agent means a live request from a person failed. The assistant could not read your page, so your content cannot appear in that answer.

rule

Best practices and common mistakes

check_circle Best practices

  • done Export a full week of logs when possible, since a single day can miss crawlers that visit on a slower schedule.
  • done Compare AI crawler hits and status codes before and after every robots.txt or firewall change.
  • done Filter your export to the hostname you care about so staging and unrelated sites do not distort the counts.
  • done Fix 403 and 429 responses served to search bots first, because those crawlers are the ones that enable citations.
  • done Check that the top requested URLs are pages you want cited, and add internal links to important pages they never reach.
  • done Save the summary numbers each month so you can show AI crawl trends to stakeholders.

cancel Common mistakes

  • close Concluding a crawler never visits after analyzing only a few hours of logs.
  • close Making security or billing decisions from the user agent string alone.
  • close Pasting logs from a CDN edge and your origin together, which double counts requests.
  • close Ignoring 3xx responses, since long redirect chains waste crawler attention.
  • close Blocking a bot in robots.txt and skipping the follow-up log check to confirm it stopped.
lightbulb

Common use cases

task_alt

Confirming that OAI-SearchBot, PerplexityBot, and Claude-SearchBot can fetch key service and product pages.

task_alt

Finding 403 or 429 responses that only bots receive because of a firewall or rate-limit rule.

task_alt

Checking whether a blocked training crawler actually stopped after a robots.txt change.

task_alt

Building a monthly AI crawl report for a US, UK, Australian, or Singaporean client site.

task_alt

Spotting crawl waste, such as bots repeatedly requesting deleted URLs or endless parameter variants.

groups

Who should use this

person Technical SEOs who need proof that AI crawlers reach important pages. person Developers and site reliability engineers investigating bot traffic and errors. person Publishers monitoring how often AI systems read their archive. person Agency reporting teams who need a simple AI crawl summary for clients. person Marketing leads who suspect a security rule is blocking AI search bots.
dictionary

Glossary

Access log
A server file recording every request with time, client address, requested path, status code, and user agent. It is the raw source this tool analyzes.
User agent
A text string a client sends to identify itself, such as GPTBot or a browser. It can be changed by the sender.
Status code
The three-digit HTTP response number. 2xx means success, 3xx redirect, 4xx client error, and 5xx server error.
Combined log format
The standard Apache and Nginx layout with IP, time, request, status, size, referrer, and user agent, which this parser reads best.
WAF
A web application firewall that filters requests before they reach your site. It can accidentally block legitimate crawlers by returning 403.
Crawl budget
The number of URLs a crawler is willing to request from a site in a period. Errors and redirects waste it.
Reverse DNS
A lookup from an IP address to a hostname, used to confirm that a crawler really belongs to the operator it claims.
Referrer
The header showing which page a visitor came from. Crawlers usually send none, which appears as a dash in logs.
quiz

Frequently asked questions

Ready for a deeper audit?

HeyLead reviews your site, competitors, and growth opportunities with no sales pressure.

Get a Free Audit arrow_forward