← Free Tools

AI Crawler Checker

See whether ChatGPT, Claude, and Google's AI crawlers can reach and read your page. No sign-up, no email.

Limit of 5 checks per hour. The tool only reads public pages.
Report

Results

Fetched as GPTBot from my server, not OpenAI's. Sites that verify crawlers may treat the real GPTBot differently.
Checked just now

AI crawler access (robots.txt)

What each crawler is allowed to do on this URL, based on the site's robots.txt file.
CrawlerUsed forStatusMatching rule

Blocking a training crawler like GPTBot doesn't remove you from ChatGPT search, which uses OAI-SearchBot. Many sites block training and allow search on purpose. Blocking Google-Extended doesn't affect AI Overviews, which run on Googlebot.

What an AI crawler can read

Text found in the raw page, before any JavaScript runs. Most AI crawlers read pages this way.

Page details

Structured data (schema)

Schema types found on the page.
Something blocked or missing?
I can fix crawler access and the rest of your AI search setup.
AI Search Optimization Book a Discovery Call
How It Works

What the AI Crawler Checker Tests

Robots.txt rules for each AI bot

Reads your robots.txt and applies the rules to each crawler for the exact URL you entered, the same way the bots do. You see which rule decides each result.

Whether the crawler is turned away

Requests the page as GPTBot. A 403, 402, or 429 response usually means a firewall or bot blocker is stopping AI crawlers before they read anything.

Content visible without JavaScript

Many AI crawlers don't run JavaScript. If your text only appears after scripts load, they may see a nearly empty page.

Which sites have this problem? ↓

Schema and page basics

Lists the structured data on the page along with the title, description, and meta robots tag, which all shape how AI tools summarize you.

FAQs

AI Crawler FAQs

It depends on your goals. GPTBot collects data for training OpenAI's models. Blocking it doesn't keep you out of ChatGPT search, which uses OAI-SearchBot. If you want to show up in AI answers, keep the search crawlers allowed at minimum.
Firewalls, CDNs, and hosting bot protection can block crawlers before robots.txt is ever read. If the robots.txt check passes but the fetch fails, look at your host's or CDN's bot settings.
Add a group for that crawler, for example User-agent: OAI-SearchBot followed by Allow: /, and make sure no broader rule blocks it. If you're on WordPress, an SEO plugin like Rank Math can edit robots.txt for you.
Most don't, or only in limited cases. Googlebot does render JavaScript, but crawlers like GPTBot and ClaudeBot generally read the raw page. Content that loads only through scripts may never be seen.
Custom React or Vue apps, older Lovable projects, and many apps from AI builders like Bolt load their text with JavaScript, so AI crawlers can see a nearly empty page. Newer Lovable projects render on the server, and older ones serve a pre-rendered version to verified crawlers only, which this tool can't see. WordPress, Webflow, Squarespace, and Shopify usually include the text in the page itself, though widgets like reviews or pricing tables can still be invisible.
No. The tool fetches the page, builds the report, and keeps nothing beyond a short-lived counter used for the hourly limit.

Want More Than a Crawl Check?

Crawler access is step one. I help brands get mentioned and cited in ChatGPT, Claude, and Google's AI Overviews.

See AI Search Optimization